Attaching to ozones3_scm_1, ozones3_datanode_3, ozones3_datanode_2, ozones3_datanode_1, ozones3_om_1, ozones3_s3g_1 scm_1 | 2019-10-15 03:14:37,962 INFO server.StorageContainerManagerStarter: STARTUP_MSG: scm_1 | /************************************************************ scm_1 | STARTUP_MSG: Starting StorageContainerManager datanode_2 | 2019-10-15 03:14:37,846 [main] INFO - STARTUP_MSG: s3g_1 | 2019-10-15 03:14:37,966 INFO hdfs.DFSUtil: Starting Web-server for s3gateway at: http://0.0.0.0:9878 datanode_2 | /************************************************************ s3g_1 | 2019-10-15 03:14:38,002 INFO util.log: Logging initialized @1069ms scm_1 | STARTUP_MSG: host = ef408b9ad96b/172.18.0.7 datanode_2 | STARTUP_MSG: Starting HddsDatanodeService s3g_1 | 2019-10-15 03:14:38,119 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_3 | 2019-10-15 03:14:37,804 [main] INFO - STARTUP_MSG: datanode_1 | 2019-10-15 03:14:37,699 [main] INFO - STARTUP_MSG: scm_1 | STARTUP_MSG: args = [--init] datanode_2 | STARTUP_MSG: host = b2e6e9267224/172.18.0.5 datanode_3 | /************************************************************ s3g_1 | 2019-10-15 03:14:38,149 INFO http.HttpRequestLog: Http request log for http.requests.s3gateway is not defined datanode_1 | /************************************************************ scm_1 | STARTUP_MSG: version = 3.2.0 datanode_3 | STARTUP_MSG: Starting HddsDatanodeService datanode_2 | STARTUP_MSG: args = [] s3g_1 | 2019-10-15 03:14:38,158 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) datanode_1 | STARTUP_MSG: Starting HddsDatanodeService scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar datanode_3 | STARTUP_MSG: host = 074e53f4bc48/172.18.0.6 s3g_1 | 2019-10-15 03:14:38,161 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context s3gateway datanode_2 | STARTUP_MSG: version = 3.2.0 om_1 | 2019-10-15 03:14:36,407 [main] INFO - STARTUP_MSG: datanode_1 | STARTUP_MSG: host = 2783c458fa57/172.18.0.4 scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_3 | STARTUP_MSG: args = [] datanode_2 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar s3g_1 | 2019-10-15 03:14:38,161 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static om_1 | /************************************************************ datanode_1 | STARTUP_MSG: args = [] scm_1 | STARTUP_MSG: java = 11.0.3 datanode_3 | STARTUP_MSG: version = 3.2.0 datanode_2 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z s3g_1 | 2019-10-15 03:14:38,161 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs datanode_1 | STARTUP_MSG: version = 3.2.0 datanode_3 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar datanode_2 | STARTUP_MSG: java = 11.0.3 om_1 | STARTUP_MSG: Starting OzoneManager s3g_1 | 2019-10-15 03:14:38,181 [main] INFO - Starting Ozone S3 gateway datanode_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/jaxb-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/activation-1.1.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaxb-core-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-datanode-0.5.0-SNAPSHOT.jar datanode_3 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_2 | ************************************************************/ om_1 | STARTUP_MSG: host = a68850063c35/172.18.0.3 s3g_1 | 2019-10-15 03:14:38,194 INFO http.HttpServer2: Jetty bound to port 9878 datanode_3 | STARTUP_MSG: java = 11.0.3 om_1 | STARTUP_MSG: args = [--init] scm_1 | ************************************************************/ s3g_1 | 2019-10-15 03:14:38,197 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_3 | ************************************************************/ om_1 | STARTUP_MSG: version = 3.2.0 scm_1 | 2019-10-15 03:14:37,971 INFO server.StorageContainerManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT] s3g_1 | 2019-10-15 03:14:38,236 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@1b66c0fb{/logs,file:///var/log/hadoop/,AVAILABLE} datanode_2 | 2019-10-15 03:14:37,863 [main] INFO - registered UNIX signal handlers for [TERM, HUP, INT] datanode_1 | STARTUP_MSG: java = 11.0.3 datanode_3 | 2019-10-15 03:14:37,818 [main] INFO - registered UNIX signal handlers for [TERM, HUP, INT] om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar scm_1 | 2019-10-15 03:14:38,142 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. s3g_1 | 2019-10-15 03:14:38,237 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@465232e9{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-s3gateway-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} datanode_1 | ************************************************************/ datanode_2 | 2019-10-15 03:14:38,154 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties datanode_3 | 2019-10-15 03:14:38,090 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z scm_1 | SCM initialization succeeded.Current cluster id for sd=/data/metadata/scm;cid=CID-ec872722-5ac5-498d-9538-21c6f7f12d58 s3g_1 | ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 datanode_2 | 2019-10-15 03:14:38,347 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). datanode_1 | 2019-10-15 03:14:37,713 [main] INFO - registered UNIX signal handlers for [TERM, HUP, INT] scm_1 | 2019-10-15 03:14:38,188 INFO server.StorageContainerManagerStarter: SHUTDOWN_MSG: s3g_1 | WARNING: An illegal reflective access operation has occurred datanode_3 | 2019-10-15 03:14:38,266 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). datanode_2 | 2019-10-15 03:14:38,348 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started datanode_1 | 2019-10-15 03:14:37,978 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties scm_1 | /************************************************************ om_1 | STARTUP_MSG: java = 11.0.3 s3g_1 | WARNING: Illegal reflective access by org.jboss.classfilewriter.ClassFile$1 (file:/opt/hadoop/share/ozone/lib/weld-servlet-2.4.7.Final.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int) datanode_2 | 2019-10-15 03:14:38,527 [main] INFO - HddsDatanodeService host:b2e6e9267224 ip:172.18.0.5 datanode_3 | 2019-10-15 03:14:38,266 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started datanode_1 | 2019-10-15 03:14:38,159 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). scm_1 | SHUTDOWN_MSG: Shutting down StorageContainerManager at ef408b9ad96b/172.18.0.7 s3g_1 | WARNING: Please consider reporting this to the maintainers of org.jboss.classfilewriter.ClassFile$1 datanode_3 | 2019-10-15 03:14:38,445 [main] INFO - HddsDatanodeService host:074e53f4bc48 ip:172.18.0.6 datanode_2 | 2019-10-15 03:14:38,568 [main] INFO - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048 om_1 | ************************************************************/ datanode_1 | 2019-10-15 03:14:38,160 INFO impl.MetricsSystemImpl: HddsDatanode metrics system started scm_1 | ************************************************************/ s3g_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations datanode_3 | 2019-10-15 03:14:38,486 [main] INFO - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048 datanode_2 | 2019-10-15 03:14:38,570 [main] INFO - Added Volume : /data/hdds/hdds to VolumeSet datanode_1 | 2019-10-15 03:14:38,343 [main] INFO - HddsDatanodeService host:2783c458fa57 ip:172.18.0.4 scm_1 | 2019-10-15 03:14:39,267 INFO server.StorageContainerManagerStarter: STARTUP_MSG: s3g_1 | WARNING: All illegal access operations will be denied in a future release datanode_3 | 2019-10-15 03:14:38,489 [main] INFO - Added Volume : /data/hdds/hdds to VolumeSet om_1 | 2019-10-15 03:14:36,417 [main] INFO - registered UNIX signal handlers for [TERM, HUP, INT] datanode_2 | 2019-10-15 03:14:38,578 [main] INFO - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@5cde6747 datanode_1 | 2019-10-15 03:14:38,393 [main] INFO - Creating Volume: /data/hdds/hdds of storage type : DISK and capacity : 10908285698048 scm_1 | /************************************************************ s3g_1 | Oct 15, 2019 3:14:40 AM org.glassfish.jersey.internal.Errors logErrors datanode_3 | 2019-10-15 03:14:38,498 [main] INFO - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@1199fe66 om_1 | 2019-10-15 03:14:37,359 [main] INFO - Configuration either no ozone.om.address set. Falling back to the default OM address om/172.18.0.3:9862 datanode_2 | 2019-10-15 03:14:38,600 [main] INFO - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@5cde6747 datanode_1 | 2019-10-15 03:14:38,396 [main] INFO - Added Volume : /data/hdds/hdds to VolumeSet scm_1 | STARTUP_MSG: Starting StorageContainerManager s3g_1 | WARNING: The following warnings have been detected: WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.get(java.lang.String,java.lang.String,java.lang.String,int,java.lang.String,java.io.InputStream) throws java.io.IOException,org.apache.hadoop.ozone.s3.exception.OS3Exception, should not consume any entity. om_1 | 2019-10-15 03:14:37,359 [main] INFO - OM Service ID is not set. Setting it to the default ID: omServiceIdDefault datanode_1 | 2019-10-15 03:14:38,409 [main] INFO - Scheduling a check for org.apache.hadoop.ozone.container.common.volume.HddsVolume@3ed242a4 datanode_2 | 2019-10-15 03:14:39,486 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs datanode_3 | 2019-10-15 03:14:38,519 [main] INFO - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1199fe66 scm_1 | STARTUP_MSG: host = ef408b9ad96b/172.18.0.7 s3g_1 | om_1 | 2019-10-15 03:14:37,365 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_1 | 2019-10-15 03:14:38,437 [main] INFO - Scheduled health check for volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@3ed242a4 datanode_2 | 2019-10-15 03:14:39,521 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default) datanode_3 | 2019-10-15 03:14:39,433 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs scm_1 | STARTUP_MSG: args = [] s3g_1 | 2019-10-15 03:14:40,990 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@151732fb{/,file:///tmp/jetty-0.0.0.0-9878-s3gateway-_-any-10865552185819346622.dir/webapp/,AVAILABLE}{/s3gateway} om_1 | 2019-10-15 03:14:38,620 INFO ipc.Client: Retrying connect to server: scm/172.18.0.7:9863. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) datanode_1 | 2019-10-15 03:14:39,360 WARN scm.HddsServerUtil: Storage directory for Ratis is not configured. It is a good idea to map this to an SSD disk. Falling back to ozone.metadata.dirs datanode_2 | 2019-10-15 03:14:39,755 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom) datanode_3 | 2019-10-15 03:14:39,465 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default) scm_1 | STARTUP_MSG: version = 3.2.0 s3g_1 | 2019-10-15 03:14:40,996 INFO server.AbstractConnector: Started ServerConnector@1bbddada{HTTP/1.1,[http/1.1]}{0.0.0.0:9878} om_1 | 2019-10-15 03:14:39,621 INFO ipc.Client: Retrying connect to server: scm/172.18.0.7:9863. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) datanode_1 | 2019-10-15 03:14:39,394 INFO impl.RaftServerProxy: raft.rpc.type = GRPC (default) datanode_2 | 2019-10-15 03:14:39,757 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom) datanode_3 | 2019-10-15 03:14:39,673 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom) s3g_1 | 2019-10-15 03:14:40,997 INFO server.Server: Started @4067ms scm_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hamcrest-all-1.3.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar om_1 | 2019-10-15 03:14:40,622 INFO ipc.Client: Retrying connect to server: scm/172.18.0.7:9863. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) datanode_1 | 2019-10-15 03:14:39,590 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.port = 9858 (custom) datanode_2 | 2019-10-15 03:14:39,760 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_3 | 2019-10-15 03:14:39,674 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom) s3g_1 | 2019-10-15 03:14:41,000 INFO server.BaseHttpServer: HTTP server of S3GATEWAY is listening at http://0.0.0.0:9878 om_1 | 2019-10-15 03:14:41,623 INFO ipc.Client: Retrying connect to server: scm/172.18.0.7:9863. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) datanode_1 | 2019-10-15 03:14:39,596 INFO server.GrpcService: raft.grpc.message.size.max = 33570816 (custom) scm_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_2 | 2019-10-15 03:14:39,761 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default) datanode_3 | 2019-10-15 03:14:39,676 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) s3g_1 | 2019-10-15 03:15:47,589 [qtp212683148-104] INFO - Location is /bucket-64893 om_1 | OM initialization succeeded.Current cluster id for sd=/data/metadata/om;cid=CID-ec872722-5ac5-498d-9538-21c6f7f12d58 datanode_1 | 2019-10-15 03:14:39,599 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | STARTUP_MSG: java = 11.0.3 datanode_2 | 2019-10-15 03:14:39,761 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_3 | 2019-10-15 03:14:39,676 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default) s3g_1 | 2019-10-15 03:15:54,019 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties om_1 | 2019-10-15 03:14:41,902 [shutdown-hook-0] INFO - SHUTDOWN_MSG: datanode_1 | 2019-10-15 03:14:39,600 INFO server.GrpcService: raft.grpc.flow.control.window = 1MB (=1048576) (default) scm_1 | ************************************************************/ datanode_2 | 2019-10-15 03:14:39,905 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_3 | 2019-10-15 03:14:39,677 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) s3g_1 | 2019-10-15 03:15:54,067 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). om_1 | /************************************************************ datanode_1 | 2019-10-15 03:14:39,601 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) scm_1 | 2019-10-15 03:14:39,275 INFO server.StorageContainerManagerStarter: registered UNIX signal handlers for [TERM, HUP, INT] datanode_2 | 2019-10-15 03:14:40,072 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 datanode_3 | 2019-10-15 03:14:39,841 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) s3g_1 | 2019-10-15 03:15:54,068 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started om_1 | SHUTDOWN_MSG: Shutting down OzoneManager at a68850063c35/172.18.0.3 datanode_1 | 2019-10-15 03:14:39,771 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) scm_1 | 2019-10-15 03:14:39,497 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_2 | 2019-10-15 03:14:40,104 INFO util.log: Logging initialized @3078ms datanode_3 | 2019-10-15 03:14:40,005 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 s3g_1 | 2019-10-15 03:15:54,070 WARN impl.MetricsSystemImpl: Sink prometheus already exists! datanode_1 | 2019-10-15 03:14:39,929 INFO hdfs.DFSUtil: Starting Web-server for hddsDatanode at: http://0.0.0.0:9882 om_1 | ************************************************************/ scm_1 | 2019-10-15 03:14:39,508 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_2 | 2019-10-15 03:14:40,211 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_3 | 2019-10-15 03:14:40,028 INFO util.log: Logging initialized @3002ms s3g_1 | 2019-10-15 03:18:07,016 [qtp212683148-3946] INFO - Location is /bucket-47263 datanode_1 | 2019-10-15 03:14:39,951 INFO util.log: Logging initialized @3034ms om_1 | 2019-10-15 03:14:42,862 [main] INFO - STARTUP_MSG: scm_1 | 2019-10-15 03:14:39,540 INFO util.log: Logging initialized @1167ms datanode_2 | 2019-10-15 03:14:40,214 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined datanode_3 | 2019-10-15 03:14:40,115 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. s3g_1 | 2019-10-15 03:18:07,689 [qtp212683148-3945] INFO - Location is /bucket-47263 datanode_1 | 2019-10-15 03:14:40,047 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. om_1 | /************************************************************ scm_1 | 2019-10-15 03:14:39,656 INFO db.DBStoreBuilder: using custom profile for table: deletedBlocks datanode_2 | 2019-10-15 03:14:40,223 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) datanode_3 | 2019-10-15 03:14:40,118 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined s3g_1 | 2019-10-15 03:18:07,894 [qtp212683148-3690] WARN - Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.GroupMismatchException: 749b19c7-0772-44d2-8efd-0664e6aa0748: group-D5F35BC43A13 not found. on the pipeline Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN]. The last committed block length is 0, uncommitted data length is 29 retry count 0 datanode_1 | 2019-10-15 03:14:40,051 INFO http.HttpRequestLog: Http request log for http.requests.hddsDatanode is not defined scm_1 | 2019-10-15 03:14:39,656 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:deletedBlocks om_1 | STARTUP_MSG: Starting OzoneManager datanode_2 | 2019-10-15 03:14:40,226 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode datanode_3 | 2019-10-15 03:14:40,125 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) s3g_1 | 2019-10-15 03:18:07,894 [qtp212683148-105] WARN - Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-D5F35BC43A13 not found. on the pipeline Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN]. The last committed block length is 0, uncommitted data length is 29 retry count 0 datanode_1 | 2019-10-15 03:14:40,061 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) om_1 | STARTUP_MSG: host = a68850063c35/172.18.0.3 scm_1 | 2019-10-15 03:14:39,656 INFO db.DBStoreBuilder: using custom profile for table: validCerts datanode_2 | 2019-10-15 03:14:40,226 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs datanode_3 | 2019-10-15 03:14:40,127 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode datanode_1 | 2019-10-15 03:14:40,065 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hddsDatanode om_1 | STARTUP_MSG: args = [] scm_1 | 2019-10-15 03:14:39,656 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:validCerts datanode_2 | 2019-10-15 03:14:40,226 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static datanode_3 | 2019-10-15 03:14:40,127 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static datanode_1 | 2019-10-15 03:14:40,065 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs om_1 | STARTUP_MSG: version = 3.2.0 s3g_1 | 2019-10-15 03:18:12,992 [qtp212683148-105] INFO - Location is /bucket-83438 scm_1 | 2019-10-15 03:14:39,656 INFO db.DBStoreBuilder: using custom profile for table: revokedCerts datanode_2 | 2019-10-15 03:14:40,263 INFO http.HttpServer2: Jetty bound to port 9882 datanode_3 | 2019-10-15 03:14:40,127 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs datanode_1 | 2019-10-15 03:14:40,065 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static s3g_1 | 2019-10-15 03:18:14,392 [qtp212683148-30] ERROR - Exception occurred in headBucket scm_1 | 2019-10-15 03:14:39,657 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:revokedCerts om_1 | STARTUP_MSG: classpath = /etc/hadoop:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/sqlite-jdbc-3.25.2.jar:/opt/hadoop/share/ozone/lib/netty-3.10.5.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/jaxb-impl-2.3.0.1.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-tools-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-netty-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.11.0.jar:/opt/hadoop/share/ozone/lib/netty-all-4.0.52.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.2.0.jar:/opt/hadoop/share/ozone/lib/jline-0.9.94.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-framework-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/metrics-ganglia-3.2.5.jar:/opt/hadoop/share/ozone/lib/libthrift-0.12.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/xz-1.0.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-config-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/zookeeper-3.4.13.jar:/opt/hadoop/share/ozone/lib/avro-1.7.7.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/guava-11.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-common-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-framework-2.12.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/json-smart-2.3.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/log4j-1.2.17.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/accessors-smart-1.2.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/curator-recipes-2.12.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/gmetric4j-1.0.7.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.31.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.2.jar:/opt/hadoop/share/ozone/lib/jaeger-core-0.34.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/ratis-proto-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/curator-client-2.12.0.jar:/opt/hadoop/share/ozone/lib/audience-annotations-0.5.0.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-6.0.1.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.5.jar:/opt/hadoop/share/ozone/lib/ratis-client-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/paranamer-2.3.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.11.0.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/gson-2.2.4.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/okio-1.13.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.5.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.16.jar:/opt/hadoop/share/ozone/lib/hadoop-ozone-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-common-0.5.0-3f446aa-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.9.9.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/picocli-3.9.6.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-0.34.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.9.9.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-docs-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.0.5.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-0.34.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-3.9.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/jettison-1.1.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jersey-json-1.19.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.25.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/ozone/lib/hadoop-hdds-client-0.5.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.31.0.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.2.0.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.2.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.3.25.v20180904.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.4.jar:/opt/hadoop/share/ozone/lib/metrics-jvm-3.2.5.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar datanode_2 | 2019-10-15 03:14:40,265 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_3 | 2019-10-15 03:14:40,157 INFO http.HttpServer2: Jetty bound to port 9882 datanode_1 | 2019-10-15 03:14:40,100 INFO http.HttpServer2: Jetty bound to port 9882 s3g_1 | org.apache.hadoop.ozone.s3.exception.OS3Exception datanode_2 | 2019-10-15 03:14:40,294 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@4228bf58{/logs,file:///var/log/hadoop/,AVAILABLE} scm_1 | 2019-10-15 03:14:39,666 INFO db.DBStoreBuilder: using custom profile for table: default om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z datanode_1 | 2019-10-15 03:14:40,102 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 s3g_1 | at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:103) scm_1 | 2019-10-15 03:14:39,667 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:default datanode_2 | 2019-10-15 03:14:40,295 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@25aeb5ac{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} datanode_3 | 2019-10-15 03:14:40,158 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 om_1 | STARTUP_MSG: java = 11.0.3 datanode_1 | 2019-10-15 03:14:40,137 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@67e13bd0{/logs,file:///var/log/hadoop/,AVAILABLE} s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getBucket(EndpointBase.java:81) scm_1 | 2019-10-15 03:14:39,668 INFO db.DBStoreBuilder: Using default options. DBProfile.DISK datanode_2 | 2019-10-15 03:14:40,367 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@358ab600{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-9188102768519995521.dir/webapp/,AVAILABLE}{/hddsDatanode} datanode_3 | 2019-10-15 03:14:40,189 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@60c8a093{/logs,file:///var/log/hadoop/,AVAILABLE} om_1 | ************************************************************/ s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.head(BucketEndpoint.java:250) scm_1 | 2019-10-15 03:14:39,863 INFO net.NodeSchemaLoader: Loading file from java.lang.CompoundEnumeration@51c693d datanode_2 | 2019-10-15 03:14:40,374 INFO server.AbstractConnector: Started ServerConnector@33956d1a{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} om_1 | 2019-10-15 03:14:42,873 [main] INFO - registered UNIX signal handlers for [TERM, HUP, INT] datanode_3 | 2019-10-15 03:14:40,190 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@75d982d3{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} datanode_1 | 2019-10-15 03:14:40,138 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@68b9834c{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) scm_1 | 2019-10-15 03:14:39,865 INFO net.NodeSchemaLoader: Loading network topology layer schema file datanode_2 | 2019-10-15 03:14:40,374 INFO server.Server: Started @3348ms datanode_3 | 2019-10-15 03:14:40,320 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@574b7f4a{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-11320206275847380078.dir/webapp/,AVAILABLE}{/hddsDatanode} om_1 | 2019-10-15 03:14:43,870 [main] INFO - Configuration either no ozone.om.address set. Falling back to the default OM address om/172.18.0.3:9862 datanode_1 | 2019-10-15 03:14:40,225 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@1f03fba0{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-17695264715968123928.dir/webapp/,AVAILABLE}{/hddsDatanode} s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) scm_1 | 2019-10-15 03:14:39,959 INFO node.SCMNodeManager: Entering startup safe mode. datanode_2 | 2019-10-15 03:14:40,379 INFO impl.MetricsSinkAdapter: Sink prometheus started datanode_3 | 2019-10-15 03:14:40,328 INFO server.AbstractConnector: Started ServerConnector@40e32762{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} om_1 | 2019-10-15 03:14:43,870 [main] INFO - OM Service ID is not set. Setting it to the default ID: omServiceIdDefault datanode_1 | 2019-10-15 03:14:40,244 INFO server.AbstractConnector: Started ServerConnector@4e1a29e2{HTTP/1.1,[http/1.1]}{0.0.0.0:9882} s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) scm_1 | 2019-10-15 03:14:40,088 INFO algorithms.ContainerPlacementPolicyFactory: Create container placement policy of type org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom datanode_3 | 2019-10-15 03:14:40,328 INFO server.Server: Started @3302ms om_1 | 2019-10-15 03:14:43,875 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_1 | 2019-10-15 03:14:40,245 INFO server.Server: Started @3329ms s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) datanode_2 | 2019-10-15 03:14:40,379 INFO impl.MetricsSystemImpl: Registered sink prometheus scm_1 | 2019-10-15 03:14:40,101 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_3 | 2019-10-15 03:14:40,334 INFO impl.MetricsSinkAdapter: Sink prometheus started s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) datanode_2 | 2019-10-15 03:14:40,381 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 scm_1 | 2019-10-15 03:14:40,176 INFO pipeline.SCMPipelineManager: No pipeline exists in current db datanode_1 | 2019-10-15 03:14:40,249 INFO impl.MetricsSinkAdapter: Sink prometheus started datanode_3 | 2019-10-15 03:14:40,334 INFO impl.MetricsSystemImpl: Registered sink prometheus om_1 | 2019-10-15 03:14:43,887 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) datanode_2 | 2019-10-15 03:14:40,396 INFO util.JvmPauseMonitor: Starting JVM pause monitor scm_1 | 2019-10-15 03:14:40,179 WARN server.ServerUtils: ozone.scm.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. datanode_1 | 2019-10-15 03:14:40,249 INFO impl.MetricsSystemImpl: Registered sink prometheus datanode_3 | 2019-10-15 03:14:40,336 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 om_1 | 2019-10-15 03:14:44,703 WARN server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) datanode_2 | 2019-10-15 03:14:40,540 [Datanode State Machine Thread - 0] INFO - DatanodeDetails is persisted to /data/datanode.id scm_1 | 2019-10-15 03:14:40,304 WARN events.EventQueue: No event handler registered for event TypedEvent{payloadType=SafeModeStatus, name='SafeModeStatus'} datanode_1 | 2019-10-15 03:14:40,251 INFO server.BaseHttpServer: HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882 datanode_3 | 2019-10-15 03:14:40,368 INFO util.JvmPauseMonitor: Starting JVM pause monitor om_1 | 2019-10-15 03:14:44,725 INFO util.log: Logging initialized @2647ms s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) datanode_2 | 2019-10-15 03:14:42,514 [Datanode State Machine Thread - 0] INFO - Attempting to start container services. scm_1 | 2019-10-15 03:14:40,810 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. datanode_1 | 2019-10-15 03:14:40,267 INFO util.JvmPauseMonitor: Starting JVM pause monitor datanode_3 | 2019-10-15 03:14:40,539 [Datanode State Machine Thread - 0] INFO - DatanodeDetails is persisted to /data/datanode.id om_1 | 2019-10-15 03:14:44,840 INFO db.DBStoreBuilder: using custom profile for table: userTable s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) datanode_2 | 2019-10-15 03:14:42,517 [Datanode State Machine Thread - 0] INFO - Background container scanner has been disabled. scm_1 | 2019-10-15 03:14:40,829 INFO ipc.Server: Starting Socket Reader #1 for port 9861 datanode_1 | 2019-10-15 03:14:40,424 [Datanode State Machine Thread - 0] INFO - DatanodeDetails is persisted to /data/datanode.id datanode_3 | 2019-10-15 03:14:42,514 [Datanode State Machine Thread - 0] INFO - Attempting to start container services. om_1 | 2019-10-15 03:14:44,840 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:userTable s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) scm_1 | 2019-10-15 03:14:40,861 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. datanode_1 | 2019-10-15 03:14:42,448 [Datanode State Machine Thread - 0] INFO - Attempting to start container services. datanode_3 | 2019-10-15 03:14:42,517 [Datanode State Machine Thread - 0] INFO - Background container scanner has been disabled. om_1 | 2019-10-15 03:14:44,840 INFO db.DBStoreBuilder: using custom profile for table: volumeTable s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) scm_1 | 2019-10-15 03:14:40,863 INFO ipc.Server: Starting Socket Reader #1 for port 9863 datanode_2 | 2019-10-15 03:14:42,517 [Datanode State Machine Thread - 0] INFO - Starting XceiverServerRatis 749b19c7-0772-44d2-8efd-0664e6aa0748 at port 9858 datanode_1 | 2019-10-15 03:14:42,451 [Datanode State Machine Thread - 0] INFO - Background container scanner has been disabled. datanode_3 | 2019-10-15 03:14:42,517 [Datanode State Machine Thread - 0] INFO - Starting XceiverServerRatis cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at port 9858 om_1 | 2019-10-15 03:14:44,840 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:volumeTable s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) datanode_2 | 2019-10-15 03:14:42,536 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: start RPC server datanode_1 | 2019-10-15 03:14:42,451 [Datanode State Machine Thread - 0] INFO - Starting XceiverServerRatis 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c at port 9858 scm_1 | 2019-10-15 03:14:40,879 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. datanode_3 | 2019-10-15 03:14:42,536 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start RPC server om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: using custom profile for table: bucketTable s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) datanode_2 | 2019-10-15 03:14:42,648 INFO server.GrpcService: 749b19c7-0772-44d2-8efd-0664e6aa0748: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 datanode_1 | 2019-10-15 03:14:42,472 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start RPC server scm_1 | 2019-10-15 03:14:40,880 INFO ipc.Server: Starting Socket Reader #1 for port 9860 datanode_3 | 2019-10-15 03:14:42,653 INFO server.GrpcService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:bucketTable datanode_2 | WARNING: An illegal reflective access operation has occurred datanode_1 | 2019-10-15 03:14:42,583 INFO server.GrpcService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858 scm_1 | 2019-10-15 03:14:40,910 INFO hdfs.DFSUtil: Starting Web-server for scm at: http://0.0.0.0:9876 datanode_3 | WARNING: An illegal reflective access operation has occurred s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: using custom profile for table: keyTable datanode_2 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address datanode_1 | WARNING: An illegal reflective access operation has occurred scm_1 | 2019-10-15 03:14:41,109 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:keyTable datanode_2 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil datanode_1 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address scm_1 | 2019-10-15 03:14:41,122 INFO http.HttpRequestLog: Http request log for http.requests.scm is not defined s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) datanode_3 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: using custom profile for table: deletedTable datanode_2 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations datanode_1 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil scm_1 | 2019-10-15 03:14:41,128 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) datanode_3 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil om_1 | 2019-10-15 03:14:44,841 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:deletedTable s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) datanode_2 | WARNING: All illegal access operations will be denied in a future release datanode_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations scm_1 | 2019-10-15 03:14:41,130 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context scm datanode_3 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: using custom profile for table: openKeyTable s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) datanode_2 | 2019-10-15 03:14:48,103 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: addNew group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-B866F3D9F4B7:java.util.concurrent.CompletableFuture@4c7b6c02[Not completed] datanode_1 | WARNING: All illegal access operations will be denied in a future release scm_1 | 2019-10-15 03:14:41,130 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:openKeyTable s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) datanode_3 | WARNING: All illegal access operations will be denied in a future release datanode_2 | 2019-10-15 03:14:48,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748: new RaftServerImpl for group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized scm_1 | 2019-10-15 03:14:41,130 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs datanode_1 | 2019-10-15 03:14:45,210 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: addNew group-6E9FEAD04B3B:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858] returns group-6E9FEAD04B3B:java.util.concurrent.CompletableFuture@22ab72a7[Not completed] om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: using custom profile for table: s3Table s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) datanode_3 | 2019-10-15 03:14:48,068 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: addNew group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-B866F3D9F4B7:java.util.concurrent.CompletableFuture@49d29973[Not completed] datanode_2 | 2019-10-15 03:14:48,127 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | 2019-10-15 03:14:41,154 INFO server.StorageContainerManager: StorageContainerLocationProtocol RPC server is listening at /0.0.0.0:9860 datanode_1 | 2019-10-15 03:14:45,224 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: new RaftServerImpl for group-6E9FEAD04B3B:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858] with ContainerStateMachine:uninitialized om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:s3Table s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) datanode_3 | 2019-10-15 03:14:48,091 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: new RaftServerImpl for group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized datanode_2 | 2019-10-15 03:14:48,127 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) scm_1 | 2019-10-15 03:14:41,210 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties datanode_1 | 2019-10-15 03:14:45,226 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: using custom profile for table: multipartInfoTable s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) datanode_3 | 2019-10-15 03:14:48,093 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | 2019-10-15 03:14:41,245 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). datanode_1 | 2019-10-15 03:14:45,226 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_2 | 2019-10-15 03:14:48,127 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:multipartInfoTable s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) datanode_3 | 2019-10-15 03:14:48,094 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) scm_1 | 2019-10-15 03:14:41,245 INFO impl.MetricsSystemImpl: StorageContainerManager metrics system started datanode_1 | 2019-10-15 03:14:45,226 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_2 | 2019-10-15 03:14:48,128 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: using custom profile for table: dTokenTable s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) datanode_3 | 2019-10-15 03:14:48,094 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) scm_1 | 2019-10-15 03:14:41,487 INFO server.SCMClientProtocolServer: RPC server for Client is listening at /0.0.0.0:9860 datanode_1 | 2019-10-15 03:14:45,227 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_2 | 2019-10-15 03:14:48,129 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) om_1 | 2019-10-15 03:14:44,842 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:dTokenTable s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) datanode_3 | 2019-10-15 03:14:48,095 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) scm_1 | 2019-10-15 03:14:41,488 INFO ipc.Server: IPC Server Responder: starting datanode_1 | 2019-10-15 03:14:45,227 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-15 03:14:48,136 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= om_1 | 2019-10-15 03:14:44,843 INFO db.DBStoreBuilder: using custom profile for table: s3SecretTable s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) datanode_3 | 2019-10-15 03:14:48,096 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) scm_1 | 2019-10-15 03:14:41,488 INFO ipc.Server: IPC Server listener on 9860: starting datanode_1 | 2019-10-15 03:14:45,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null, confs= datanode_2 | 2019-10-15 03:14:48,136 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) om_1 | 2019-10-15 03:14:44,843 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:s3SecretTable s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) datanode_3 | 2019-10-15 03:14:48,104 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_1 | 2019-10-15 03:14:45,233 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | 2019-10-15 03:14:48,141 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) om_1 | 2019-10-15 03:14:44,843 INFO db.DBStoreBuilder: using custom profile for table: prefixTable scm_1 | 2019-10-15 03:14:41,492 INFO server.StorageContainerManager: ScmBlockLocationProtocol RPC server is listening at /0.0.0.0:9863 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) datanode_1 | 2019-10-15 03:14:45,237 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_3 | 2019-10-15 03:14:48,105 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | 2019-10-15 03:14:48,143 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 does not exist. Creating ... om_1 | 2019-10-15 03:14:44,843 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:prefixTable scm_1 | 2019-10-15 03:14:41,493 INFO server.SCMBlockProtocolServer: RPC server for Block Protocol is listening at /0.0.0.0:9863 s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) datanode_1 | 2019-10-15 03:14:45,238 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/e385a798-f2c4-4eec-b54b-6e9fead04b3b does not exist. Creating ... datanode_3 | 2019-10-15 03:14:48,110 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) om_1 | 2019-10-15 03:14:44,853 INFO db.DBStoreBuilder: using custom profile for table: default scm_1 | 2019-10-15 03:14:41,493 INFO ipc.Server: IPC Server Responder: starting s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) datanode_1 | 2019-10-15 03:14:46,255 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/e385a798-f2c4-4eec-b54b-6e9fead04b3b/in_use.lock acquired by nodename 7@2783c458fa57 om_1 | 2019-10-15 03:14:44,853 INFO db.DBStoreBuilder: Using default column profile:DBProfile.DISK for Table:default datanode_3 | 2019-10-15 03:14:48,111 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 does not exist. Creating ... datanode_2 | 2019-10-15 03:14:50,780 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: addNew group-180984EA4EAD:[749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858] returns group-180984EA4EAD:java.util.concurrent.CompletableFuture@3bef3b70[Not completed] scm_1 | 2019-10-15 03:14:41,494 INFO ipc.Server: IPC Server listener on 9863: starting s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | 2019-10-15 03:14:44,854 INFO db.DBStoreBuilder: Using default options. DBProfile.DISK datanode_2 | 2019-10-15 03:14:53,625 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/in_use.lock acquired by nodename 7@b2e6e9267224 scm_1 | 2019-10-15 03:14:41,496 INFO server.StorageContainerManager: ScmDatanodeProtocl RPC server is listening at /0.0.0.0:9861 s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) datanode_3 | 2019-10-15 03:14:53,624 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/in_use.lock acquired by nodename 7@074e53f4bc48 om_1 | 2019-10-15 03:14:53,927 INFO ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 2000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. datanode_2 | 2019-10-15 03:14:53,645 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 has been successfully formatted. datanode_1 | 2019-10-15 03:14:47,763 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: addNew group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-B866F3D9F4B7:java.util.concurrent.CompletableFuture@38b1a554[Not completed] scm_1 | 2019-10-15 03:14:41,496 INFO server.SCMDatanodeProtocolServer: RPC server for DataNodes is listening at /0.0.0.0:9861 om_1 | 2019-10-15 03:14:53,936 INFO ipc.Server: Starting Socket Reader #1 for port 9862 datanode_3 | 2019-10-15 03:14:53,645 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 has been successfully formatted. datanode_2 | 2019-10-15 03:14:53,652 [pool-8-thread-1] INFO - group-B866F3D9F4B7: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) scm_1 | 2019-10-15 03:14:41,497 INFO ipc.Server: IPC Server Responder: starting datanode_1 | 2019-10-15 03:14:50,118 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/e385a798-f2c4-4eec-b54b-6e9fead04b3b has been successfully formatted. datanode_3 | 2019-10-15 03:14:53,650 [pool-8-thread-1] INFO - group-B866F3D9F4B7: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) om_1 | 2019-10-15 03:14:53,976 [main] INFO - OzoneManager RPC server is listening at om/172.18.0.3:9862 datanode_2 | 2019-10-15 03:14:53,653 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) scm_1 | 2019-10-15 03:14:41,497 INFO ipc.Server: IPC Server listener on 9861: starting datanode_1 | 2019-10-15 03:14:50,124 [pool-8-thread-1] INFO - group-6E9FEAD04B3B: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_3 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) om_1 | 2019-10-15 03:14:54,065 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties datanode_2 | 2019-10-15 03:14:53,657 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) scm_1 | 2019-10-15 03:14:41,503 INFO http.HttpServer2: Jetty bound to port 9876 datanode_1 | 2019-10-15 03:14:50,130 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_3 | 2019-10-15 03:14:53,653 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) om_1 | 2019-10-15 03:14:54,114 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). datanode_2 | 2019-10-15 03:14:53,666 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) scm_1 | 2019-10-15 03:14:41,505 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_1 | 2019-10-15 03:14:50,133 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_3 | 2019-10-15 03:14:53,659 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) om_1 | 2019-10-15 03:14:54,114 INFO impl.MetricsSystemImpl: OzoneManager metrics system started datanode_2 | 2019-10-15 03:14:53,666 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) scm_1 | 2019-10-15 03:14:41,545 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6ae7deac{/logs,file:///var/log/hadoop/,AVAILABLE} datanode_1 | 2019-10-15 03:14:50,138 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_3 | 2019-10-15 03:14:53,659 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) om_1 | 2019-10-15 03:14:54,159 INFO ipc.Server: IPC Server Responder: starting datanode_2 | 2019-10-15 03:14:53,669 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) datanode_1 | 2019-10-15 03:14:50,138 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_3 | 2019-10-15 03:14:53,662 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) scm_1 | 2019-10-15 03:14:41,545 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@21fdfefc{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-hdds-server-scm-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} om_1 | 2019-10-15 03:14:54,162 INFO ipc.Server: IPC Server listener on 9862: starting datanode_2 | 2019-10-15 03:14:53,671 WARN impl.MetricsSystemImpl: HddsDatanode metrics system already initialized! s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) datanode_1 | 2019-10-15 03:14:50,140 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_3 | 2019-10-15 03:14:53,664 WARN impl.MetricsSystemImpl: HddsDatanode metrics system already initialized! scm_1 | 2019-10-15 03:14:41,638 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@1907874b{/,file:///tmp/jetty-0.0.0.0-9876-scm-_-any-1785914471691836244.dir/webapp/,AVAILABLE}{/scm} om_1 | 2019-10-15 03:14:54,194 INFO hdfs.DFSUtil: Starting Web-server for ozoneManager at: http://0.0.0.0:9874 datanode_2 | 2019-10-15 03:14:53,679 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl datanode_1 | 2019-10-15 03:14:50,142 WARN impl.MetricsSystemImpl: HddsDatanode metrics system already initialized! datanode_3 | 2019-10-15 03:14:53,671 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl scm_1 | 2019-10-15 03:14:41,647 INFO server.AbstractConnector: Started ServerConnector@48528634{HTTP/1.1,[http/1.1]}{0.0.0.0:9876} s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) om_1 | 2019-10-15 03:14:54,316 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. datanode_2 | 2019-10-15 03:14:53,725 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_1 | 2019-10-15 03:14:50,148 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl datanode_3 | 2019-10-15 03:14:53,715 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) om_1 | 2019-10-15 03:14:54,319 INFO http.HttpRequestLog: Http request log for http.requests.ozoneManager is not defined scm_1 | 2019-10-15 03:14:41,647 INFO server.Server: Started @3274ms datanode_2 | 2019-10-15 03:14:53,731 INFO segmented.SegmentedRaftLogWorker: new 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 datanode_1 | 2019-10-15 03:14:50,191 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_3 | 2019-10-15 03:14:53,721 INFO segmented.SegmentedRaftLogWorker: new cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) om_1 | 2019-10-15 03:14:54,327 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) scm_1 | 2019-10-15 03:14:41,652 INFO impl.MetricsSinkAdapter: Sink prometheus started datanode_2 | 2019-10-15 03:14:53,731 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_1 | 2019-10-15 03:14:50,197 INFO segmented.SegmentedRaftLogWorker: new 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/e385a798-f2c4-4eec-b54b-6e9fead04b3b datanode_3 | 2019-10-15 03:14:53,721 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) om_1 | 2019-10-15 03:14:54,331 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context ozoneManager scm_1 | 2019-10-15 03:14:41,652 INFO impl.MetricsSystemImpl: Registered sink prometheus datanode_1 | 2019-10-15 03:14:50,197 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_3 | 2019-10-15 03:14:53,721 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | 2019-10-15 03:14:53,732 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) om_1 | 2019-10-15 03:14:54,331 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs scm_1 | 2019-10-15 03:14:41,654 INFO server.BaseHttpServer: HTTP server of SCM is listening at http://0.0.0.0:9876 datanode_1 | 2019-10-15 03:14:50,198 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_3 | 2019-10-15 03:14:53,722 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-15 03:14:53,733 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) om_1 | 2019-10-15 03:14:54,331 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static datanode_1 | 2019-10-15 03:14:50,199 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_3 | 2019-10-15 03:14:53,723 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) scm_1 | 2019-10-15 03:14:41,669 INFO util.JvmPauseMonitor: Starting JVM pause monitor datanode_2 | 2019-10-15 03:14:53,734 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) om_1 | 2019-10-15 03:14:54,356 INFO http.HttpServer2: Jetty bound to port 9874 datanode_1 | 2019-10-15 03:14:50,200 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_3 | 2019-10-15 03:14:53,723 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) datanode_2 | 2019-10-15 03:14:53,734 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) scm_1 | 2019-10-15 03:14:44,305 INFO net.NetworkTopology: Added a new node: /default-rack/5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c om_1 | 2019-10-15 03:14:54,357 INFO server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-04T21:11:46Z, git hash: 3ce520221d0240229c862b122d2b06c12a625732 datanode_1 | 2019-10-15 03:14:50,200 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_3 | 2019-10-15 03:14:53,724 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) datanode_2 | 2019-10-15 03:14:53,735 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) scm_1 | 2019-10-15 03:14:44,306 INFO node.SCMNodeManager: Registered Data node : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null} om_1 | 2019-10-15 03:14:54,388 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@169268a7{/logs,file:///var/log/hadoop/,AVAILABLE} datanode_1 | 2019-10-15 03:14:50,201 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_3 | 2019-10-15 03:14:53,725 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) datanode_2 | 2019-10-15 03:14:53,736 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) scm_1 | 2019-10-15 03:14:44,309 INFO safemode.SCMSafeModeManager: SCM in safe mode. 1 DataNodes registered, 1 required. om_1 | 2019-10-15 03:14:54,389 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@313f8301{/static,jar:file:/opt/hadoop/share/ozone/lib/hadoop-ozone-ozone-manager-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} datanode_1 | 2019-10-15 03:14:50,202 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_3 | 2019-10-15 03:14:53,725 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | 2019-10-15 03:14:53,736 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) scm_1 | 2019-10-15 03:14:44,310 INFO safemode.SCMSafeModeManager: ScmSafeModeManager, all rules are successfully validated om_1 | 2019-10-15 03:14:54,485 INFO handler.ContextHandler: Started o.e.j.w.WebAppContext@2b5c4f17{/,file:///tmp/jetty-0.0.0.0-9874-ozoneManager-_-any-6426382283726805819.dir/webapp/,AVAILABLE}{/ozoneManager} datanode_1 | 2019-10-15 03:14:50,202 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) datanode_3 | 2019-10-15 03:14:53,725 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) scm_1 | 2019-10-15 03:14:44,310 INFO safemode.SCMSafeModeManager: SCM exiting safe mode. datanode_2 | 2019-10-15 03:14:53,736 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) om_1 | 2019-10-15 03:14:54,490 INFO server.AbstractConnector: Started ServerConnector@6848a051{HTTP/1.1,[http/1.1]}{0.0.0.0:9874} s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) datanode_3 | 2019-10-15 03:14:53,735 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | 2019-10-15 03:14:50,202 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) scm_1 | 2019-10-15 03:14:44,396 INFO net.NetworkTopology: Added a new node: /default-rack/cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | 2019-10-15 03:14:53,746 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) om_1 | 2019-10-15 03:14:54,490 INFO server.Server: Started @12412ms s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) datanode_1 | 2019-10-15 03:14:50,213 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_3 | 2019-10-15 03:14:53,739 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 scm_1 | 2019-10-15 03:14:44,397 INFO node.SCMNodeManager: Registered Data node : cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_2 | 2019-10-15 03:14:53,751 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 om_1 | 2019-10-15 03:14:54,493 INFO impl.MetricsSinkAdapter: Sink prometheus started s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) datanode_1 | 2019-10-15 03:14:50,218 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_3 | 2019-10-15 03:14:53,745 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) scm_1 | 2019-10-15 03:14:44,416 INFO net.NetworkTopology: Added a new node: /default-rack/749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:53,756 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) om_1 | 2019-10-15 03:14:54,493 INFO impl.MetricsSystemImpl: Registered sink prometheus s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) datanode_1 | 2019-10-15 03:14:50,224 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) scm_1 | 2019-10-15 03:14:44,416 INFO node.SCMNodeManager: Registered Data node : 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_3 | 2019-10-15 03:14:53,745 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | 2019-10-15 03:14:53,757 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) om_1 | 2019-10-15 03:14:54,495 INFO server.BaseHttpServer: HTTP server of OZONEMANAGER is listening at http://0.0.0.0:9874 s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) datanode_1 | 2019-10-15 03:14:50,226 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) scm_1 | WARNING: An illegal reflective access operation has occurred datanode_3 | 2019-10-15 03:14:53,746 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | 2019-10-15 03:14:53,757 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) om_1 | 2019-10-15 03:14:59,005 [IPC Server handler 1 on 9862] INFO - created volume:vol-0-07672 for user:hadoop s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:14:50,226 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) scm_1 | WARNING: Illegal reflective access by org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil (file:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-0.2.0.jar) to field java.nio.Buffer.address datanode_3 | 2019-10-15 03:14:53,746 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-15 03:14:53,758 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) om_1 | 2019-10-15 03:14:59,051 [IPC Server handler 3 on 9862] INFO - created volume:vol-1-71066 for user:hadoop s3g_1 | 2019-10-15 03:18:19,364 [qtp212683148-104] INFO - Location is /bucket-77614 datanode_1 | 2019-10-15 03:14:50,227 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) scm_1 | WARNING: Please consider reporting this to the maintainers of org.apache.ratis.thirdparty.com.google.protobuf.UnsafeUtil datanode_3 | 2019-10-15 03:14:53,780 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:53,786 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: addNew group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-D5F35BC43A13:java.util.concurrent.CompletableFuture@4768453d[Not completed] om_1 | 2019-10-15 03:14:59,062 [IPC Server handler 4 on 9862] INFO - created volume:vol-2-40494 for user:hadoop s3g_1 | 2019-10-15 03:18:25,221 [qtp212683148-3690] INFO - Location is /bucket-19451 datanode_1 | 2019-10-15 03:14:50,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: new RaftServerImpl for group-B866F3D9F4B7:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized scm_1 | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations datanode_3 | 2019-10-15 03:14:53,781 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from null to FOLLOWER at term 0 for startAsFollower om_1 | 2019-10-15 03:14:59,073 [IPC Server handler 7 on 9862] INFO - created volume:vol-3-47882 for user:hadoop datanode_2 | 2019-10-15 03:14:53,796 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748: new RaftServerImpl for group-180984EA4EAD:[749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858] with ContainerStateMachine:uninitialized s3g_1 | 2019-10-15 03:18:44,760 [qtp212683148-27] ERROR - Error in Complete Multipart Upload Request for bucket: bucket-19451, key: multipartKey2 datanode_1 | 2019-10-15 03:14:50,274 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | WARNING: All illegal access operations will be denied in a future release datanode_3 | 2019-10-15 03:14:53,782 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:53,796 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) s3g_1 | ENTITY_TOO_SMALL org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: Entity too small: volume: s3408a471635b80160c579a897a992d8edbucket: bucket-19451key: multipartKey2 om_1 | 2019-10-15 03:14:59,081 [IPC Server handler 9 on 9862] INFO - created volume:vol-4-94782 for user:hadoop datanode_1 | 2019-10-15 03:14:50,274 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_3 | 2019-10-15 03:14:53,788 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | 2019-10-15 03:14:53,796 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) scm_1 | 2019-10-15 03:14:47,716 ERROR pipeline.RatisPipelineProvider: Failed invoke Ratis rpc org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider$$Lambda$173/0x00000008403ca840@74136ea3 for 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) datanode_1 | 2019-10-15 03:14:50,274 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) om_1 | 2019-10-15 03:18:07,683 [IPC Server handler 5 on 9862] ERROR - S3Bucket Creation Failed for userName: 408a471635b80160c579a897a992d8ed, s3BucketName bucket-47263, VolumeName s3408a471635b80160c579a897a992d8ed datanode_3 | 2019-10-15 03:14:53,792 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: addNew group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-D5F35BC43A13:java.util.concurrent.CompletableFuture@27d82664[Not completed] datanode_2 | 2019-10-15 03:14:53,796 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) scm_1 | org.apache.ratis.protocol.TimeoutIOException: deadline exceeded after 2958249690ns s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB.java:1104) datanode_1 | 2019-10-15 03:14:50,274 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) om_1 | 2019-10-15 03:18:44,755 [IPC Server handler 12 on 9862] ERROR - MultipartUpload: /s3408a471635b80160c579a897a992d8ed/bucket-19451/multipartKey2Part number: 1size 6 is less than minimum part size 5242880 datanode_3 | 2019-10-15 03:14:53,805 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms scm_1 | at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:82) datanode_2 | 2019-10-15 03:14:53,796 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) datanode_1 | 2019-10-15 03:14:50,274 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) om_1 | 2019-10-15 03:18:44,758 [IPC Server handler 12 on 9862] ERROR - MultipartUpload Complete request failed for Key: multipartKey2 in Volume/Bucket s3408a471635b80160c579a897a992d8ed/bucket-19451 datanode_3 | 2019-10-15 03:14:53,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:75) datanode_2 | 2019-10-15 03:14:53,796 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | 2019-10-15 03:14:50,274 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) om_1 | ENTITY_TOO_SMALL org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: Entity too small: volume: s3408a471635b80160c579a897a992d8edbucket: bucket-19451key: multipartKey2 datanode_3 | 2019-10-15 03:14:53,807 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:184) datanode_2 | 2019-10-15 03:14:53,796 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: ConfigurationManager, init=-1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null, confs= datanode_1 | 2019-10-15 03:14:50,275 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:209) datanode_3 | 2019-10-15 03:14:53,809 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderElection scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:153) datanode_2 | 2019-10-15 03:14:53,797 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_1 | 2019-10-15 03:14:50,275 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) datanode_3 | 2019-10-15 03:14:53,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: new RaftServerImpl for group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized scm_1 | at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:94) datanode_2 | 2019-10-15 03:14:53,797 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) datanode_1 | 2019-10-15 03:14:50,275 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 does not exist. Creating ... om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) datanode_3 | 2019-10-15 03:14:53,815 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:278) datanode_2 | 2019-10-15 03:14:53,797 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/43c1a53c-72d8-41b4-9784-180984ea4ead does not exist. Creating ... s3g_1 | at com.sun.proxy.$Proxy84.completeMultipartUpload(Unknown Source) datanode_1 | 2019-10-15 03:14:53,643 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/in_use.lock acquired by nodename 7@2783c458fa57 om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) datanode_3 | 2019-10-15 03:14:53,815 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.groupAdd(RaftClientImpl.java:205) datanode_2 | 2019-10-15 03:14:53,810 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/43c1a53c-72d8-41b4-9784-180984ea4ead/in_use.lock acquired by nodename 7@b2e6e9267224 s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.completeMultipartUpload(RpcClient.java:883) datanode_1 | 2019-10-15 03:14:53,647 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 has been successfully formatted. om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) datanode_3 | 2019-10-15 03:14:53,815 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$initializePipeline$4(RatisPipelineProvider.java:198) datanode_2 | 2019-10-15 03:14:53,824 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/43c1a53c-72d8-41b4-9784-180984ea4ead has been successfully formatted. s3g_1 | at org.apache.hadoop.ozone.client.OzoneBucket.completeMultipartUpload(OzoneBucket.java:445) datanode_1 | 2019-10-15 03:14:53,648 [pool-8-thread-1] INFO - group-B866F3D9F4B7: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) datanode_3 | 2019-10-15 03:14:53,815 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$null$5(RatisPipelineProvider.java:233) s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.completeMultipartUpload(ObjectEndpoint.java:498) datanode_2 | 2019-10-15 03:14:53,825 [pool-8-thread-1] INFO - group-180984EA4EAD: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_1 | 2019-10-15 03:14:53,648 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) datanode_3 | 2019-10-15 03:14:53,815 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_1 | 2019-10-15 03:14:53,649 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) datanode_3 | 2019-10-15 03:14:53,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= scm_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_1 | 2019-10-15 03:14:53,649 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) datanode_3 | 2019-10-15 03:14:53,816 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) scm_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_1 | 2019-10-15 03:14:53,649 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) datanode_3 | 2019-10-15 03:14:53,816 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_1 | 2019-10-15 03:14:53,649 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) datanode_3 | 2019-10-15 03:14:53,816 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 does not exist. Creating ... scm_1 | at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_1 | 2019-10-15 03:14:53,649 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_3 | 2019-10-15 03:14:53,824 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13/in_use.lock acquired by nodename 7@074e53f4bc48 s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) datanode_2 | 2019-10-15 03:14:53,825 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_1 | 2019-10-15 03:14:53,649 INFO segmented.SegmentedRaftLogWorker: new 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7 om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408) datanode_3 | 2019-10-15 03:14:53,824 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) datanode_2 | 2019-10-15 03:14:53,825 INFO segmented.SegmentedRaftLogWorker: new 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/43c1a53c-72d8-41b4-9784-180984ea4ead datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) datanode_3 | 2019-10-15 03:14:53,845 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 has been successfully formatted. datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) om_1 | 2019-10-15 03:18:52,204 [IPC Server handler 2 on 9862] ERROR - MultipartUpload Complete request failed for Key: multipartKey3 in Volume/Bucket s3408a471635b80160c579a897a992d8ed/bucket-19451 scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) datanode_3 | 2019-10-15 03:14:53,846 [pool-8-thread-1] INFO - group-D5F35BC43A13: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) om_1 | MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s3408a471635b80160c579a897a992d8edbucket: bucket-19451key: multipartKey3 scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) datanode_3 | 2019-10-15 03:14:53,852 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCompleteRequest.validateAndUpdateCache(S3MultipartUploadCompleteRequest.java:195) scm_1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_3 | 2019-10-15 03:14:53,853 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) scm_1 | at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) scm_1 | at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661) datanode_3 | 2019-10-15 03:14:53,853 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_1 | 2019-10-15 03:14:53,650 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$callRatisRpc$6(RatisPipelineProvider.java:227) datanode_1 | 2019-10-15 03:14:53,651 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_3 | 2019-10-15 03:14:53,853 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) scm_1 | at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) datanode_1 | 2019-10-15 03:14:53,651 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_3 | 2019-10-15 03:14:53,854 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_1 | 2019-10-15 03:14:53,651 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_3 | 2019-10-15 03:14:53,854 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) datanode_1 | 2019-10-15 03:14:53,651 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_3 | 2019-10-15 03:14:53,854 INFO segmented.SegmentedRaftLogWorker: new cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) datanode_3 | 2019-10-15 03:14:53,854 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) datanode_1 | 2019-10-15 03:14:53,652 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) datanode_2 | 2019-10-15 03:14:53,826 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_3 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) datanode_2 | 2019-10-15 03:14:53,827 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_3 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_1 | 2019-10-15 03:14:53,652 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) scm_1 | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) datanode_2 | 2019-10-15 03:14:53,827 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_3 | 2019-10-15 03:14:53,857 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_1 | 2019-10-15 03:14:53,653 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) scm_1 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2958249690ns s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) datanode_3 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_1 | 2019-10-15 03:14:53,653 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | 2019-10-15 03:14:53,827 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) datanode_3 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_1 | 2019-10-15 03:14:53,653 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-15 03:14:53,827 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) datanode_3 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_1 | 2019-10-15 03:14:53,659 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null datanode_2 | 2019-10-15 03:14:53,828 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) om_1 | 2019-10-15 03:19:03,651 [IPC Server handler 17 on 9862] ERROR - Abort Multipart request is failed for KeyName multipartKey5 in VolumeName/Bucket s3408a471635b80160c579a897a992d8ed/bucket-19451 datanode_3 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_1 | 2019-10-15 03:14:53,660 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | 2019-10-15 03:14:53,835 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748: new RaftServerImpl for group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized scm_1 | at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:274) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: Abort Multipart Upload Failed: volume: s3408a471635b80160c579a897a992d8edbucket: bucket-19451key: multipartKey5 datanode_3 | 2019-10-15 03:14:53,859 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_1 | 2019-10-15 03:14:53,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:53,836 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$3(GrpcClientProtocolClient.java:155) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) om_1 | at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadAbortRequest.validateAndUpdateCache(S3MultipartUploadAbortRequest.java:116) datanode_3 | 2019-10-15 03:14:53,859 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | 2019-10-15 03:14:53,669 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-6E9FEAD04B3B,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:53,836 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:182) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) datanode_3 | 2019-10-15 03:14:53,860 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_1 | 2019-10-15 03:14:53,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:53,836 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) scm_1 | ... 26 more s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) datanode_3 | 2019-10-15 03:14:53,861 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_1 | 2019-10-15 03:14:53,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from null to FOLLOWER at term 0 for startAsFollower scm_1 | 2019-10-15 03:14:50,735 ERROR pipeline.RatisPipelineProvider: Failed invoke Ratis rpc org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider$$Lambda$173/0x00000008403ca840@7cc585d0 for 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) datanode_3 | 2019-10-15 03:14:53,861 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | 2019-10-15 03:14:53,836 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_1 | 2019-10-15 03:14:53,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | org.apache.ratis.protocol.TimeoutIOException: deadline exceeded after 2998417020ns s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) datanode_3 | 2019-10-15 03:14:53,862 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | 2019-10-15 03:14:53,836 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | 2019-10-15 03:14:53,700 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:82) s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) datanode_3 | 2019-10-15 03:14:53,862 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-15 03:14:53,836 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_1 | 2019-10-15 03:14:53,711 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:48ms scm_1 | at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:75) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) datanode_3 | 2019-10-15 03:14:53,871 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:53,837 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:184) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) datanode_3 | 2019-10-15 03:14:53,871 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_1 | 2019-10-15 03:14:53,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:53,837 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:153) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) datanode_3 | 2019-10-15 03:14:53,872 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:53,714 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) datanode_2 | 2019-10-15 03:14:53,837 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 does not exist. Creating ... scm_1 | at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:94) s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) datanode_3 | 2019-10-15 03:14:53,872 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_1 | 2019-10-15 03:14:53,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) datanode_2 | 2019-10-15 03:14:53,850 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13/in_use.lock acquired by nodename 7@b2e6e9267224 scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:278) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) datanode_3 | 2019-10-15 03:14:53,981 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7:t1, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,748 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-LeaderElection1: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) datanode_2 | 2019-10-15 03:14:53,854 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 has been successfully formatted. scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.groupAdd(RaftClientImpl.java:205) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) datanode_3 | 2019-10-15 03:14:53,997 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: begin an election at term 2 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,750 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) datanode_2 | 2019-10-15 03:14:53,854 [pool-8-thread-1] INFO - group-D5F35BC43A13: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$initializePipeline$4(RatisPipelineProvider.java:198) datanode_3 | 2019-10-15 03:14:54,000 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_1 | 2019-10-15 03:14:53,750 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: changes role from CANDIDATE to LEADER at term 1 for changeToLeader om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$null$5(RatisPipelineProvider.java:233) datanode_3 | 2019-10-15 03:14:54,012 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:53,751 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: change Leader from null to 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c at term 1 for becomeLeader, leader elected after 3620ms om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) datanode_3 | 2019-10-15 03:14:54,012 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_1 | 2019-10-15 03:14:53,755 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) om_1 | 2019-10-15 03:19:09,175 [IPC Server handler 4 on 9862] ERROR - ALLOCATE_KEY failed for Key: multipartKey in volume/bucket:s3408a471635b80160c579a897a992d8ed/bucket-19451 datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) scm_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) datanode_3 | 2019-10-15 03:14:54,012 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderElection datanode_1 | 2019-10-15 03:14:53,755 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) om_1 | NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload is with specified uploadId random datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) datanode_3 | 2019-10-15 03:14:54,029 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection2: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) datanode_1 | 2019-10-15 03:14:53,758 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareMultipartKeyInfo(OMKeyRequest.java:471) datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) datanode_3 | 2019-10-15 03:14:54,051 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: Election TIMEOUT; received 1 response(s) [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t2] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7:t2, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) datanode_1 | 2019-10-15 03:14:53,761 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.prepareKeyInfo(OMKeyRequest.java:423) datanode_2 | 2019-10-15 03:14:53,855 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) datanode_3 | 2019-10-15 03:14:54,070 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: begin an election at term 3 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,761 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) scm_1 | at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) om_1 | at org.apache.hadoop.ozone.om.request.key.OMKeyCreateRequest.validateAndUpdateCache(OMKeyCreateRequest.java:179) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) datanode_3 | 2019-10-15 03:14:54,093 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection2: Election REJECTED; received 2 response(s) [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t1, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13:t1, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:53,856 INFO segmented.SegmentedRaftLogWorker: new 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 datanode_1 | 2019-10-15 03:14:53,762 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:219) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) datanode_3 | 2019-10-15 03:14:54,094 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_1 | 2019-10-15 03:14:53,787 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: addNew group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-D5F35BC43A13:java.util.concurrent.CompletableFuture@6f9b57c4[Not completed] om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:134) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408) datanode_3 | 2019-10-15 03:14:54,095 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_1 | 2019-10-15 03:14:53,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderState om_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) datanode_3 | 2019-10-15 03:14:54,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_1 | 2019-10-15 03:14:53,791 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: new RaftServerImpl for group-D5F35BC43A13:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized scm_1 | at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736) om_1 | at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:100) s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) datanode_1 | 2019-10-15 03:14:53,791 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) om_1 | at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java) datanode_3 | 2019-10-15 03:14:54,107 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-LeaderElection1: Election PASSED; received 2 response(s) [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t3, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:OK-t3] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7:t3, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_1 | 2019-10-15 03:14:53,791 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) om_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) datanode_3 | 2019-10-15 03:14:54,109 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:53,792 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) scm_1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) om_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) datanode_1 | 2019-10-15 03:14:53,792 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) datanode_3 | 2019-10-15 03:14:54,110 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from CANDIDATE to LEADER at term 3 for changeToLeader scm_1 | at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) datanode_1 | 2019-10-15 03:14:53,792 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) datanode_3 | 2019-10-15 03:14:54,112 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: change Leader from null to cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at term 3 for becomeLeader, leader elected after 462ms scm_1 | at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661) datanode_2 | 2019-10-15 03:14:53,856 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) om_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) datanode_1 | 2019-10-15 03:14:53,792 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$callRatisRpc$6(RatisPipelineProvider.java:227) datanode_2 | 2019-10-15 03:14:53,857 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) om_1 | at java.base/java.security.AccessController.doPrivileged(Native Method) datanode_1 | 2019-10-15 03:14:53,792 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_3 | 2019-10-15 03:14:54,119 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) scm_1 | at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) datanode_2 | 2019-10-15 03:14:53,857 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) om_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423) datanode_1 | 2019-10-15 03:14:53,793 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_3 | 2019-10-15 03:14:54,119 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) s3g_1 | 2019-10-15 03:18:52,205 [qtp212683148-30] ERROR - Error in Complete Multipart Upload Request for bucket: bucket-19451, key: multipartKey3 scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_2 | 2019-10-15 03:14:53,857 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 om_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) datanode_1 | 2019-10-15 03:14:53,793 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 does not exist. Creating ... s3g_1 | MISMATCH_MULTIPART_LIST org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multipart Upload Failed: volume: s3408a471635b80160c579a897a992d8edbucket: bucket-19451key: multipartKey3 datanode_3 | 2019-10-15 03:14:54,124 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) datanode_2 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) om_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) datanode_1 | 2019-10-15 03:14:53,796 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13/in_use.lock acquired by nodename 7@2783c458fa57 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:732) datanode_3 | 2019-10-15 03:14:54,128 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) datanode_2 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_1 | 2019-10-15 03:14:53,806 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker: Starting segment from index:0 s3g_1 | at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.completeMultipartUpload(OzoneManagerProtocolClientSideTranslatorPB.java:1104) datanode_3 | 2019-10-15 03:14:54,128 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) datanode_2 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_1 | 2019-10-15 03:14:53,810 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 has been successfully formatted. s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) datanode_3 | 2019-10-15 03:14:54,129 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) scm_1 | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) datanode_2 | 2019-10-15 03:14:53,858 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_1 | 2019-10-15 03:14:53,810 [pool-8-thread-1] INFO - group-D5F35BC43A13: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) scm_1 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2998417020ns datanode_3 | 2019-10-15 03:14:54,149 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_2 | 2019-10-15 03:14:53,863 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) datanode_3 | 2019-10-15 03:14:54,150 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | 2019-10-15 03:14:53,864 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) datanode_3 | 2019-10-15 03:14:54,151 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_2 | 2019-10-15 03:14:53,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) s3g_1 | at org.apache.hadoop.hdds.tracing.TraceAllMethod.invoke(TraceAllMethod.java:66) datanode_3 | 2019-10-15 03:14:54,153 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_2 | 2019-10-15 03:14:53,870 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:274) s3g_1 | at com.sun.proxy.$Proxy84.completeMultipartUpload(Unknown Source) datanode_3 | 2019-10-15 03:14:54,154 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_2 | 2019-10-15 03:14:53,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: start as a follower, conf=-1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$3(GrpcClientProtocolClient.java:155) s3g_1 | at org.apache.hadoop.ozone.client.rpc.RpcClient.completeMultipartUpload(RpcClient.java:883) datanode_3 | 2019-10-15 03:14:54,154 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-15 03:14:53,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: changes role from null to FOLLOWER at term 0 for startAsFollower scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:182) s3g_1 | at org.apache.hadoop.ozone.client.OzoneBucket.completeMultipartUpload(OzoneBucket.java:445) datanode_3 | 2019-10-15 03:14:54,159 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_2 | 2019-10-15 03:14:53,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState s3g_1 | at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.completeMultipartUpload(ObjectEndpoint.java:498) scm_1 | ... 26 more datanode_1 | 2019-10-15 03:14:53,811 INFO segmented.SegmentedRaftLogWorker: new 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13 datanode_2 | 2019-10-15 03:14:53,889 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-180984EA4EAD,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:54,160 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) scm_1 | 2019-10-15 03:14:50,738 ERROR pipeline.RatisPipelineProvider: Failed invoke Ratis rpc org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider$$Lambda$173/0x00000008403ca840@7cc585d0 for cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_2 | 2019-10-15 03:14:53,893 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,160 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) scm_1 | org.apache.ratis.protocol.TimeoutIOException: deadline exceeded after 2997040464ns datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_3 | 2019-10-15 03:14:54,160 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 2 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:53,893 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from null to FOLLOWER at term 0 for startAsFollower s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) scm_1 | at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:82) datanode_1 | 2019-10-15 03:14:53,811 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_3 | 2019-10-15 03:14:54,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:53,893 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566) scm_1 | at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:75) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_3 | 2019-10-15 03:14:54,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:53,894 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:184) s3g_1 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_3 | 2019-10-15 03:14:54,160 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_2 | 2019-10-15 03:14:53,931 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:153) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_3 | 2019-10-15 03:14:54,161 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_2 | 2019-10-15 03:14:53,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:94) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_3 | 2019-10-15 03:14:54,161 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-15 03:14:53,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:278) s3g_1 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_3 | 2019-10-15 03:14:54,161 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.groupAdd(RaftClientImpl.java:205) s3g_1 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | 2019-10-15 03:14:53,950 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:54,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$initializePipeline$4(RatisPipelineProvider.java:198) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) datanode_1 | 2019-10-15 03:14:53,812 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_2 | 2019-10-15 03:14:53,965 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-LeaderElection1: begin an election at term 1 for -1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null datanode_3 | 2019-10-15 03:14:54,196 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: Starting segment from index:0 scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$null$5(RatisPipelineProvider.java:233) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) datanode_1 | 2019-10-15 03:14:53,813 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | 2019-10-15 03:14:53,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:54,201 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 3 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) s3g_1 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) datanode_1 | 2019-10-15 03:14:53,813 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_2 | 2019-10-15 03:14:53,967 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_3 | 2019-10-15 03:14:54,201 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) datanode_1 | 2019-10-15 03:14:53,813 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | 2019-10-15 03:14:53,967 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: change Leader from null to 749b19c7-0772-44d2-8efd-0664e6aa0748 at term 1 for becomeLeader, leader elected after 142ms datanode_3 | 2019-10-15 03:14:54,201 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) datanode_1 | 2019-10-15 03:14:53,813 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) datanode_2 | 2019-10-15 03:14:53,972 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) datanode_3 | 2019-10-15 03:14:54,201 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:53,813 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) datanode_2 | 2019-10-15 03:14:53,973 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) scm_1 | at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) datanode_1 | 2019-10-15 03:14:53,816 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:316) datanode_2 | 2019-10-15 03:14:53,976 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_1 | 2019-10-15 03:14:53,816 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from null to FOLLOWER at term 0 for startAsFollower s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:298) datanode_2 | 2019-10-15 03:14:53,977 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:54,207 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 scm_1 | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) datanode_1 | 2019-10-15 03:14:53,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:268) datanode_2 | 2019-10-15 03:14:53,977 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_3 | 2019-10-15 03:14:54,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7- LEADER: Withhold vote from candidate 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c with term 2. State: leader=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, term=3, lastRpcElapsed=null datanode_1 | 2019-10-15 03:14:53,817 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:53,977 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection scm_1 | at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) datanode_3 | 2019-10-15 03:14:54,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7- LEADER: Withhold vote from candidate 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c with term 3. State: leader=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, term=3, lastRpcElapsed=null datanode_1 | 2019-10-15 03:14:53,819 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null at 0 datanode_2 | 2019-10-15 03:14:53,979 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) datanode_3 | 2019-10-15 03:14:54,209 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7- LEADER: Withhold vote from candidate 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c with term 1. State: leader=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, term=3, lastRpcElapsed=null datanode_1 | 2019-10-15 03:14:53,888 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:71ms, electionTimeout:71ms datanode_2 | 2019-10-15 03:14:53,983 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) scm_1 | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) datanode_3 | 2019-10-15 03:14:54,222 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:5ms datanode_1 | 2019-10-15 03:14:53,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:53,983 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) scm_1 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2997040464ns s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) datanode_3 | 2019-10-15 03:14:54,224 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:53,889 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | 2019-10-15 03:14:53,984 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) datanode_3 | 2019-10-15 03:14:54,235 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 3 for changeToCandidate datanode_1 | 2019-10-15 03:14:53,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:53,995 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderState scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) datanode_3 | 2019-10-15 03:14:54,236 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderElection datanode_1 | 2019-10-15 03:14:53,892 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | 2019-10-15 03:14:53,997 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection2: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) datanode_3 | 2019-10-15 03:14:54,240 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7- LEADER: Withhold vote from candidate 749b19c7-0772-44d2-8efd-0664e6aa0748 with term 1. State: leader=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, term=3, lastRpcElapsed=null datanode_1 | 2019-10-15 03:14:53,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,014 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker: Starting segment from index:0 scm_1 | at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:274) s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) datanode_3 | 2019-10-15 03:14:54,275 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection3: begin an election at term 4 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,892 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | 2019-10-15 03:14:54,018 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 2 for recognizeCandidate:cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$3(GrpcClientProtocolClient.java:155) s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840) datanode_3 | 2019-10-15 03:14:54,292 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection3: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13:t4, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:53,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:54,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:182) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1780) datanode_3 | 2019-10-15 03:14:54,303 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: change Leader from cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 to null at term 4 for updateCurrentTerm datanode_2 | 2019-10-15 03:14:54,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | ... 16 more datanode_1 | 2019-10-15 03:14:53,895 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,303 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from LEADER to FOLLOWER at term 4 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c s3g_1 | at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) scm_1 | 2019-10-15 03:14:50,736 ERROR pipeline.RatisPipelineProvider: Failed invoke Ratis rpc org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider$$Lambda$173/0x00000008403ca840@7cc585d0 for 749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,030 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: set configuration 0: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null at 0 datanode_1 | 2019-10-15 03:14:53,899 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderState scm_1 | org.apache.ratis.protocol.TimeoutIOException: deadline exceeded after 2999201355ns datanode_2 | 2019-10-15 03:14:54,066 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection2: Election REJECTED; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t2, leader=null, voted=null, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) datanode_1 | 2019-10-15 03:14:53,930 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t1, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:82) datanode_2 | 2019-10-15 03:14:54,068 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_3 | 2019-10-15 03:14:54,305 WARN server.GrpcLogAppender: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException s3g_1 | at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) datanode_1 | 2019-10-15 03:14:53,949 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/e385a798-f2c4-4eec-b54b-6e9fead04b3b/current/log_inprogress_0 scm_1 | at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:75) datanode_2 | 2019-10-15 03:14:54,068 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:54,307 INFO impl.PendingRequests: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-PendingRequests: sendNotLeaderResponses datanode_1 | 2019-10-15 03:14:53,949 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: begin an election at term 2 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:184) datanode_2 | 2019-10-15 03:14:54,069 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_3 | 2019-10-15 03:14:54,307 WARN server.GrpcLogAppender: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException datanode_1 | 2019-10-15 03:14:54,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t2, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) datanode_2 | 2019-10-15 03:14:54,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:153) datanode_3 | 2019-10-15 03:14:54,310 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection3: begin an election at term 5 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:54,070 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: begin an election at term 3 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) datanode_2 | 2019-10-15 03:14:54,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1 for recognizeCandidate:cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 scm_1 | at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:94) datanode_3 | 2019-10-15 03:14:54,308 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,087 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) datanode_2 | 2019-10-15 03:14:54,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:278) datanode_3 | 2019-10-15 03:14:54,333 INFO server.GrpcLogAppender: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c-AppendLogResponseHandler: follower responses appendEntries COMPLETED s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) datanode_2 | 2019-10-15 03:14:54,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.groupAdd(RaftClientImpl.java:205) datanode_1 | 2019-10-15 03:14:54,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: begin an election at term 2 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,346 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 5 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) datanode_2 | 2019-10-15 03:14:54,090 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 3 for recognizeCandidate:cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_3 | 2019-10-15 03:14:54,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$initializePipeline$4(RatisPipelineProvider.java:198) s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513) datanode_2 | 2019-10-15 03:14:54,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,131 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t2, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$null$5(RatisPipelineProvider.java:233) s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) datanode_2 | 2019-10-15 03:14:54,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:54,153 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t3, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) datanode_1 | 2019-10-15 03:14:54,161 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: begin an election at term 3 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,090 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:54,346 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) datanode_2 | 2019-10-15 03:14:54,101 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection3: begin an election at term 2 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,355 INFO server.GrpcLogAppender: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748-AppendLogResponseHandler: follower responses appendEntries COMPLETED scm_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) datanode_1 | 2019-10-15 03:14:54,181 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: begin an election at term 4 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,102 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection3: Election REJECTED; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13:t2, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,359 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-LeaderElection3: Election REJECTED; received 2 response(s) [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t5, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t5] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13:t5, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) datanode_1 | 2019-10-15 03:14:54,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t3, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,196 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/43c1a53c-72d8-41b4-9784-180984ea4ead/current/log_inprogress_0 datanode_3 | 2019-10-15 03:14:54,361 INFO impl.FollowerInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: nextIndex: updateUnconditionally 0 -> 0 scm_1 | at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:539) datanode_1 | 2019-10-15 03:14:54,269 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: begin an election at term 4 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 5 for DISCOVERED_A_NEW_TERM scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) datanode_2 | 2019-10-15 03:14:54,210 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_1 | 2019-10-15 03:14:54,290 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t4, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,363 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderElection scm_1 | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) datanode_3 | 2019-10-15 03:14:54,363 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,305 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: begin an election at term 5 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) datanode_3 | 2019-10-15 03:14:54,364 INFO impl.FollowerInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748: nextIndex: updateUnconditionally 0 -> 0 datanode_1 | 2019-10-15 03:14:54,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection2: Election DISCOVERED_A_NEW_TERM; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t5] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t4, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,219 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 2 for changeToCandidate scm_1 | at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) datanode_3 | 2019-10-15 03:14:54,426 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: change Leader from null to 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c at term 5 for appendEntries, leader elected after 122ms datanode_1 | 2019-10-15 03:14:54,325 INFO server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Completed APPEND_ENTRIES, lastRequest: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0-t3, previous=(t:0, i:0), leaderCommit=-1, initializing? false, entries: datanode_2 | 2019-10-15 03:14:54,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection s3g_1 | at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) datanode_3 | 2019-10-15 03:14:54,426 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: change Leader from null to 749b19c7-0772-44d2-8efd-0664e6aa0748 at term 5 for appendEntries, leader elected after 574ms datanode_1 | 2019-10-15 03:14:54,327 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 5 for DISCOVERED_A_NEW_TERM scm_1 | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) datanode_2 | 2019-10-15 03:14:54,235 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: begin an election at term 3 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 datanode_1 | 2019-10-15 03:14:54,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) scm_1 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2999201355ns datanode_2 | 2019-10-15 03:14:54,253 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 4 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:54,443 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker: Starting segment from index:0 datanode_1 | 2019-10-15 03:14:54,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | 2019-10-15 03:14:54,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,331 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 5 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:54,443 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | 2019-10-15 03:14:54,253 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:54,444 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/log_inprogress_0 datanode_1 | 2019-10-15 03:14:54,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState s3g_1 | at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | 2019-10-15 03:14:54,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:54,446 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: Truncating segments toTruncate: null datanode_1 | 2019-10-15 03:14:54,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) datanode_2 | 2019-10-15 03:14:54,262 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13:t3, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:274) datanode_3 | toDelete: [(0, 0) isOpen? true, length=0, newEndIndex=-1], start index 0 datanode_1 | 2019-10-15 03:14:54,332 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) datanode_2 | 2019-10-15 03:14:54,269 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: begin an election at term 4 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$3(GrpcClientProtocolClient.java:155) datanode_3 | 2019-10-15 03:14:54,446 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: Starting segment from index:0 datanode_1 | 2019-10-15 03:14:54,356 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-LeaderElection3: Election PASSED; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:OK-t5] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t5, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null s3g_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:14:54,289 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13:t4, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:182) datanode_1 | 2019-10-15 03:14:54,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection s3g_1 | 2019-10-15 03:20:06,367 [qtp212683148-3945] INFO - Location is /bucket-52237 datanode_2 | 2019-10-15 03:14:54,305 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: begin an election at term 5 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,458 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: Deleted log file /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/log_inprogress_0 datanode_1 | 2019-10-15 03:14:54,356 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from CANDIDATE to LEADER at term 5 for changeToLeader scm_1 | ... 16 more s3g_1 | 2019-10-15 03:20:07,050 [qtp212683148-4052] INFO - Location is /destbucket-31184 datanode_2 | 2019-10-15 03:14:54,340 INFO server.GrpcServerProtocolService: 749b19c7-0772-44d2-8efd-0664e6aa0748: Completed APPEND_ENTRIES, lastRequest: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3->749b19c7-0772-44d2-8efd-0664e6aa0748#0-t3, previous=(t:0, i:0), leaderCommit=-1, initializing? false, entries: datanode_1 | 2019-10-15 03:14:54,356 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: change Leader from null to 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c at term 5 for becomeLeader, leader elected after 708ms scm_1 | 2019-10-15 03:14:53,754 ERROR pipeline.RatisPipelineProvider: Failed invoke Ratis rpc org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider$$Lambda$173/0x00000008403ca840@127d02ba for 749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:54,458 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | 2019-10-15 03:14:54,340 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 5 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c s3g_1 | 2019-10-15 03:20:32,667 [qtp212683148-3759] INFO - Location is /bucket-81526 datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) scm_1 | org.apache.ratis.protocol.TimeoutIOException: deadline exceeded after 2999150386ns datanode_3 | 2019-10-15 03:14:54,481 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: change Leader from 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c to null at term 6 for updateCurrentTerm datanode_2 | 2019-10-15 03:14:54,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState s3g_1 | 2019-10-15 03:21:15,528 [qtp212683148-105] INFO - Location is /bucket-48114 datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) scm_1 | at org.apache.ratis.grpc.GrpcUtil.tryUnwrapException(GrpcUtil.java:82) datanode_3 | 2019-10-15 03:14:54,482 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 6 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState s3g_1 | 2019-10-15 03:21:27,860 [qtp212683148-105] INFO - Location is /bucket-33585 datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) scm_1 | at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:75) datanode_3 | 2019-10-15 03:14:54,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,341 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted s3g_1 | 2019-10-15 03:22:29,780 [qtp212683148-4052] INFO - Location is /bucket-03359 scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:184) datanode_2 | 2019-10-15 03:14:54,351 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-LeaderElection4: Election PASSED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:OK-t5] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13:t5, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,483 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: change Leader from 749b19c7-0772-44d2-8efd-0664e6aa0748 to null at term 6 for updateCurrentTerm datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.groupAdd(GrpcClientProtocolClient.java:153) datanode_2 | 2019-10-15 03:14:54,351 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:54,483 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 6 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) scm_1 | at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:94) datanode_2 | 2019-10-15 03:14:54,351 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from CANDIDATE to LEADER at term 5 for changeToLeader datanode_1 | 2019-10-15 03:14:54,357 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:278) datanode_1 | 2019-10-15 03:14:54,364 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_2 | 2019-10-15 03:14:54,352 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: change Leader from null to 749b19c7-0772-44d2-8efd-0664e6aa0748 at term 5 for becomeLeader, leader elected after 497ms datanode_3 | 2019-10-15 03:14:54,483 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.ratis.client.impl.RaftClientImpl.groupAdd(RaftClientImpl.java:205) datanode_1 | 2019-10-15 03:14:54,364 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_3 | 2019-10-15 03:14:54,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,353 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$initializePipeline$4(RatisPipelineProvider.java:198) datanode_1 | 2019-10-15 03:14:54,365 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_3 | 2019-10-15 03:14:54,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,353 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$null$5(RatisPipelineProvider.java:233) datanode_1 | 2019-10-15 03:14:54,367 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_3 | 2019-10-15 03:14:54,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) datanode_2 | 2019-10-15 03:14:54,354 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) datanode_1 | 2019-10-15 03:14:54,367 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_3 | 2019-10-15 03:14:54,484 INFO server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Completed APPEND_ENTRIES, lastRequest: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#1-t5, previous=(t:0, i:0), leaderCommit=-1, initializing? false, entries: size=1, first=(t:5, i:0), CONFIGURATIONENTRY scm_1 | at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) datanode_2 | 2019-10-15 03:14:54,354 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) datanode_1 | 2019-10-15 03:14:54,368 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_3 | 2019-10-15 03:14:54,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) datanode_2 | 2019-10-15 03:14:54,356 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) datanode_3 | 2019-10-15 03:14:54,484 INFO server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Completed APPEND_ENTRIES, lastRequest: 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#1-t5, previous=(t:0, i:0), leaderCommit=-1, initializing? false, entries: size=1, first=(t:5, i:0), CONFIGURATIONENTRY datanode_2 | 2019-10-15 03:14:54,356 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) datanode_1 | 2019-10-15 03:14:54,372 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) scm_1 | at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) datanode_3 | 2019-10-15 03:14:54,496 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/log_inprogress_0 datanode_2 | 2019-10-15 03:14:54,365 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_1 | 2019-10-15 03:14:54,372 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_3 | 2019-10-15 03:14:54,496 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13/current/log_inprogress_0 datanode_2 | 2019-10-15 03:14:54,365 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408) datanode_1 | 2019-10-15 03:14:54,372 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_3 | 2019-10-15 03:14:54,516 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 7 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:54,365 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) scm_1 | at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736) datanode_1 | 2019-10-15 03:14:54,372 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_3 | 2019-10-15 03:14:54,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) datanode_1 | 2019-10-15 03:14:54,372 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_2 | 2019-10-15 03:14:54,368 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_1 | 2019-10-15 03:14:54,372 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) scm_1 | at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) datanode_3 | 2019-10-15 03:14:54,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,369 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_1 | 2019-10-15 03:14:54,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderState scm_1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) datanode_3 | 2019-10-15 03:14:54,516 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,369 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | 2019-10-15 03:14:54,376 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker: Starting segment from index:0 scm_1 | at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) datanode_3 | 2019-10-15 03:14:54,698 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 7 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,373 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_1 | 2019-10-15 03:14:54,377 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 scm_1 | at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661) datanode_3 | 2019-10-15 03:14:54,698 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,373 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_1 | 2019-10-15 03:14:54,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.lambda$callRatisRpc$6(RatisPipelineProvider.java:227) datanode_3 | 2019-10-15 03:14:54,699 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,373 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_1 | 2019-10-15 03:14:54,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) datanode_3 | 2019-10-15 03:14:54,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,373 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_1 | 2019-10-15 03:14:54,423 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 5 for changeToCandidate scm_1 | at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) datanode_3 | 2019-10-15 03:14:54,739 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 8 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,373 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_1 | 2019-10-15 03:14:54,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection scm_1 | at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) datanode_3 | 2019-10-15 03:14:54,739 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,373 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) scm_1 | at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) datanode_3 | 2019-10-15 03:14:54,739 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderState scm_1 | at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) datanode_1 | 2019-10-15 03:14:54,444 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/log_inprogress_0 datanode_3 | 2019-10-15 03:14:54,739 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,377 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker: Starting segment from index:0 scm_1 | at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) datanode_1 | 2019-10-15 03:14:54,444 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection4: begin an election at term 6 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,766 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 9 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2999150386ns datanode_1 | 2019-10-15 03:14:54,447 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: change Leader from 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c to null at term 6 for updateCurrentTerm datanode_3 | 2019-10-15 03:14:54,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,378 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:14:54,447 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from LEADER to FOLLOWER at term 6 for stepDown datanode_3 | 2019-10-15 03:14:54,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,400 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:45ms scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:14:54,447 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderState datanode_3 | 2019-10-15 03:14:54,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,400 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:14:54,460 WARN server.GrpcLogAppender: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException datanode_3 | 2019-10-15 03:14:54,819 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 10 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupManagement(AdminProtocolServiceGrpc.java:274) datanode_2 | 2019-10-15 03:14:54,400 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 5 for changeToCandidate datanode_1 | 2019-10-15 03:14:54,461 WARN server.GrpcLogAppender: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException datanode_3 | 2019-10-15 03:14:54,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.lambda$groupAdd$3(GrpcClientProtocolClient.java:155) datanode_2 | 2019-10-15 03:14:54,401 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:54,819 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,463 INFO impl.PendingRequests: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-PendingRequests: sendNotLeaderResponses scm_1 | at org.apache.ratis.grpc.client.GrpcClientProtocolClient.blockingCall(GrpcClientProtocolClient.java:182) datanode_2 | 2019-10-15 03:14:54,444 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1a183fe3-faf2-4917-9518-d5f35bc43a13/current/log_inprogress_0 datanode_3 | 2019-10-15 03:14:54,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,463 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection4: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t6, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | ... 26 more datanode_3 | 2019-10-15 03:14:54,836 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 8 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:54,444 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection5: begin an election at term 6 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:54,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | 2019-10-15 03:14:53,938 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] datanode_3 | 2019-10-15 03:14:54,836 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,461 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: change Leader from 749b19c7-0772-44d2-8efd-0664e6aa0748 to null at term 6 for updateCurrentTerm datanode_1 | 2019-10-15 03:14:54,469 INFO server.GrpcLogAppender: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748-AppendLogResponseHandler: follower responses appendEntries COMPLETED scm_1 | 2019-10-15 03:14:59,333 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 5103035a-2b13-425f-878c-94bbc5390e94, Nodes: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:STAND_ALONE, Factor:ONE, State:OPEN] datanode_3 | 2019-10-15 03:14:54,836 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:16:54,019 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: de3123d6-eac8-4e25-8c6b-366cbce63b0d, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] datanode_1 | 2019-10-15 03:14:54,486 INFO server.GrpcLogAppender: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3-AppendLogResponseHandler: follower responses appendEntries COMPLETED datanode_3 | 2019-10-15 03:14:54,836 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,466 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from LEADER to FOLLOWER at term 6 for stepDown scm_1 | 2019-10-15 03:16:54,089 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: b0c87df0-d20c-4205-986d-2aaf8832d47c, Nodes: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] datanode_1 | 2019-10-15 03:14:54,497 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection4: begin an election at term 7 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:54,960 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 9 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:54,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderState scm_1 | 2019-10-15 03:16:54,164 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 1c05e746-3997-432d-a303-357f199bef59, Nodes: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] datanode_1 | 2019-10-15 03:14:54,497 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 6 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:54,960 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,466 INFO server.GrpcServerProtocolService: 749b19c7-0772-44d2-8efd-0664e6aa0748: Completed APPEND_ENTRIES, lastRequest: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c->749b19c7-0772-44d2-8efd-0664e6aa0748#0-t5, previous=(t:0, i:0), leaderCommit=-1, initializing? false, entries: scm_1 | 2019-10-15 03:17:01,156 ERROR pipeline.PipelineActionHandler: Received pipeline action CLOSE for Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] from datanode 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}. Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120197ms datanode_1 | 2019-10-15 03:14:54,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:54,960 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | 2019-10-15 03:17:01,156 INFO pipeline.SCMPipelineManager: destroying pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] datanode_1 | 2019-10-15 03:14:54,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:54,468 WARN server.GrpcLogAppender: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException datanode_3 | 2019-10-15 03:14:54,960 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:17:01,158 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] moved to CLOSED state datanode_1 | 2019-10-15 03:14:54,498 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,468 INFO impl.PendingRequests: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-PendingRequests: sendNotLeaderResponses datanode_3 | 2019-10-15 03:14:54,996 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 11 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:54,500 INFO server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Completed APPEND_ENTRIES, lastRequest: 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#1-t5, previous=(t:0, i:0), leaderCommit=0, initializing? false, entries: size=1, first=(t:5, i:0), CONFIGURATIONENTRY scm_1 | 2019-10-15 03:17:01,159 INFO container.CloseContainerEventHandler: Close container Event triggered for container : #4 datanode_3 | 2019-10-15 03:14:54,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,518 INFO impl.FollowerInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->749b19c7-0772-44d2-8efd-0664e6aa0748: nextIndex: updateUnconditionally 0 -> 0 datanode_2 | 2019-10-15 03:14:54,468 WARN server.GrpcLogAppender: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException scm_1 | 2019-10-15 03:17:01,163 INFO container.CloseContainerEventHandler: Close container Event triggered for container : #5 datanode_3 | 2019-10-15 03:14:54,996 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,533 INFO impl.FollowerInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: nextIndex: updateUnconditionally 1 -> 0 datanode_2 | 2019-10-15 03:14:54,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | 2019-10-15 03:17:31,146 ERROR pipeline.PipelineActionHandler: Received pipeline action CLOSE for Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] from datanode 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}. Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127438ms datanode_3 | 2019-10-15 03:14:54,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,567 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection4: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t7, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t7] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t7, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | 2019-10-15 03:17:31,147 INFO pipeline.SCMPipelineManager: destroying pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_3 | 2019-10-15 03:14:55,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 10 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,568 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 7 for DISCOVERED_A_NEW_TERM scm_1 | 2019-10-15 03:17:54,235 INFO pipeline.PipelineStateManager: Created pipeline Pipeline[ Id: 9b465122-6196-4503-b052-bd8d864b3b3a, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] datanode_2 | 2019-10-15 03:14:54,486 INFO server.GrpcLogAppender: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3-AppendLogResponseHandler: follower responses appendEntries COMPLETED datanode_3 | 2019-10-15 03:14:55,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection scm_1 | 2019-10-15 03:17:55,638 ERROR pipeline.PipelineActionHandler: Received pipeline action CLOSE for Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] from datanode 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}. Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157437ms datanode_2 | 2019-10-15 03:14:54,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 6 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:55,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | 2019-10-15 03:17:55,639 INFO pipeline.SCMPipelineManager: destroying pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_2 | 2019-10-15 03:14:54,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:55,208 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | 2019-10-15 03:17:57,012 ERROR pipeline.PipelineActionHandler: Received pipeline action CLOSE for Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] from datanode 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}. Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181826ms datanode_2 | 2019-10-15 03:14:54,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:55,230 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 12 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:54,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 7 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | 2019-10-15 03:17:57,013 INFO pipeline.SCMPipelineManager: destroying pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_2 | 2019-10-15 03:14:54,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:55,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | 2019-10-15 03:18:07,227 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] removed from db datanode_2 | 2019-10-15 03:14:54,502 INFO server.GrpcLogAppender: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c-AppendLogResponseHandler: follower responses appendEntries COMPLETED datanode_3 | 2019-10-15 03:14:55,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | 2019-10-15 03:18:08,087 ERROR pipeline.PipelineActionHandler: Could not execute pipeline action=action: CLOSE datanode_2 | 2019-10-15 03:14:54,503 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection5: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t6, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t6] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t6, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:55,231 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | closePipeline { datanode_2 | 2019-10-15 03:14:54,503 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 6 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:55,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 11 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | pipelineID { datanode_3 | 2019-10-15 03:14:55,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | id: "1a183fe3-faf2-4917-9518-d5f35bc43a13" datanode_2 | 2019-10-15 03:14:54,505 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:54,699 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | } datanode_2 | 2019-10-15 03:14:54,505 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:54,739 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 8 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | reason: PIPELINE_FAILED datanode_3 | 2019-10-15 03:14:55,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,506 INFO impl.FollowerInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: nextIndex: updateUnconditionally 1 -> 0 datanode_1 | 2019-10-15 03:14:54,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | detailedReason: "5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183249ms" datanode_2 | 2019-10-15 03:14:54,507 INFO impl.FollowerInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: nextIndex: updateUnconditionally 1 -> 0 datanode_3 | 2019-10-15 03:14:55,367 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,739 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | } datanode_2 | 2019-10-15 03:14:54,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 7 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:55,433 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 13 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 {} datanode_2 | 2019-10-15 03:14:54,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:55,433 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,760 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms scm_1 | org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 not found datanode_3 | 2019-10-15 03:14:55,433 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,760 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132) datanode_2 | 2019-10-15 03:14:54,551 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:55,433 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,760 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 7 for changeToCandidate scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.getPipeline(PipelineStateManager.java:66) datanode_2 | 2019-10-15 03:14:54,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:55,451 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 12 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,760 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.getPipeline(SCMPipelineManager.java:191) datanode_2 | 2019-10-15 03:14:54,638 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_3 | 2019-10-15 03:14:55,451 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,768 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 9 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineActionHandler.onMessage(PipelineActionHandler.java:59) datanode_3 | 2019-10-15 03:14:55,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineActionHandler.onMessage(PipelineActionHandler.java:35) datanode_2 | 2019-10-15 03:14:54,638 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:55,452 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85) datanode_2 | 2019-10-15 03:14:54,639 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 6 for changeToCandidate datanode_3 | 2019-10-15 03:14:55,516 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 13 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,768 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:14:54,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:55,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,788 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection5: begin an election at term 8 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:14:54,654 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: begin an election at term 7 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:55,516 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 10 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:14:54,692 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t7, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:55,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | 2019-10-15 03:18:37,196 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_3 | 2019-10-15 03:14:55,541 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 14 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:54,806 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:55,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | 2019-10-15 03:18:37,212 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_2 | 2019-10-15 03:14:54,706 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: begin an election at term 8 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:54,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:55,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | 2019-10-15 03:18:37,227 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_2 | 2019-10-15 03:14:54,721 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t8, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:54,852 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection5: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t8, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t8] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t8, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:55,541 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: begin an election at term 9 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | 2019-10-15 03:18:37,227 ERROR pipeline.SCMPipelineManager: Destroy pipeline failed for pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_1 | 2019-10-15 03:14:54,852 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 8 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:55,585 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 15 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,763 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t9, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 not found datanode_1 | 2019-10-15 03:14:54,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:55,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,788 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: begin an election at term 10 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132) datanode_1 | 2019-10-15 03:14:54,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:55,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,817 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 8 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:322) datanode_1 | 2019-10-15 03:14:54,912 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_3 | 2019-10-15 03:14:55,585 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:54,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:107) datanode_3 | 2019-10-15 03:14:55,706 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 16 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:427) datanode_3 | 2019-10-15 03:14:55,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,817 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(SCMPipelineManager.java:413) datanode_3 | 2019-10-15 03:14:55,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,913 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 8 for changeToCandidate datanode_2 | 2019-10-15 03:14:54,834 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection6: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t10, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t10] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t10, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.lambda$finalizeAndDestroyPipeline$0(SCMPipelineManager.java:322) datanode_3 | 2019-10-15 03:14:55,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:54,834 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 10 for DISCOVERED_A_NEW_TERM scm_1 | at org.apache.hadoop.hdds.utils.Scheduler.lambda$schedule$1(Scheduler.java:68) datanode_3 | 2019-10-15 03:14:55,722 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 14 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:54,942 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection6: begin an election at term 9 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,835 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection scm_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:14:55,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,979 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection6: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t9, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t9] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t9, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:54,835 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:14:55,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,979 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 9 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:54,960 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms scm_1 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) datanode_3 | 2019-10-15 03:14:55,722 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:54,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:14:54,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:54,961 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 10 for changeToCandidate datanode_3 | 2019-10-15 03:14:55,768 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 15 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:14:54,998 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 11 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:55,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:54,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:14:54,966 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 9 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:55,769 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:54,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState scm_1 | 2019-10-15 03:18:54,194 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 1 blocks datanode_2 | 2019-10-15 03:14:54,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:55,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:54,999 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:18:54,195 INFO block.BlockManagerImpl: Deleting blocks conID: 7 locID: 102964232677359747 bcsId: 0 datanode_3 | 2019-10-15 03:14:55,868 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 16 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,168 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms scm_1 | 2019-10-15 03:19:01,672 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_3 | 2019-10-15 03:14:55,868 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | 2019-10-15 03:19:01,690 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_3 | 2019-10-15 03:14:55,868 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:19:01,705 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_3 | 2019-10-15 03:14:55,868 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:54,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:55,169 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 9 for changeToCandidate scm_1 | 2019-10-15 03:19:01,706 ERROR pipeline.SCMPipelineManager: Destroy pipeline failed for pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_3 | 2019-10-15 03:14:55,902 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 17 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:54,966 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:55,902 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:54,975 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection7: begin an election at term 11 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 not found datanode_3 | 2019-10-15 03:14:55,902 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,184 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection7: begin an election at term 10 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,014 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection7: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t11, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t11] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t11, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132) datanode_3 | 2019-10-15 03:14:55,902 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 12 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:322) datanode_3 | 2019-10-15 03:14:55,956 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 17 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,015 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 11 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:55,956 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:55,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:55,956 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:107) datanode_1 | 2019-10-15 03:14:55,232 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:55,956 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:427) datanode_1 | 2019-10-15 03:14:55,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection7: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t10, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t10] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t10, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,192 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | 2019-10-15 03:14:56,115 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 18 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,245 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 10 for DISCOVERED_A_NEW_TERM scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(SCMPipelineManager.java:413) datanode_2 | 2019-10-15 03:14:55,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:56,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,245 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.lambda$finalizeAndDestroyPipeline$0(SCMPipelineManager.java:322) datanode_2 | 2019-10-15 03:14:55,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 11 for changeToCandidate datanode_1 | 2019-10-15 03:14:55,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:56,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at org.apache.hadoop.hdds.utils.Scheduler.lambda$schedule$1(Scheduler.java:68) datanode_2 | 2019-10-15 03:14:55,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:55,326 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:79ms datanode_3 | 2019-10-15 03:14:56,115 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | 2019-10-15 03:14:55,205 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection8: begin an election at term 12 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:14:56,185 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 19 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,207 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 10 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 10 for changeToCandidate scm_1 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) datanode_2 | 2019-10-15 03:14:55,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:56,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:14:55,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:55,341 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection8: begin an election at term 11 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:56,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:14:55,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection8: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t11, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t11] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t11, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:56,185 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:14:55,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 11 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:56,203 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 18 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | 2019-10-15 03:19:03,054 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_2 | 2019-10-15 03:14:55,245 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection8: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t12, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t12] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t12, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:56,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState scm_1 | 2019-10-15 03:19:03,073 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_1 | 2019-10-15 03:14:55,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:55,246 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 12 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:56,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState scm_1 | 2019-10-15 03:19:03,089 WARN pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 dn=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null} datanode_1 | 2019-10-15 03:14:55,386 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:55,246 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:56,203 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:19:03,089 ERROR pipeline.SCMPipelineManager: Destroy pipeline failed for pipeline:Pipeline[ Id: 1a183fe3-faf2-4917-9518-d5f35bc43a13, Nodes: 749b19c7-0772-44d2-8efd-0664e6aa0748{ip: 172.18.0.5, host: ozones3_datanode_2.ozones3_default, networkLocation: /default-rack, certSerialId: null}5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c{ip: 172.18.0.4, host: ozones3_datanode_1.ozones3_default, networkLocation: /default-rack, certSerialId: null}cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3{ip: 172.18.0.6, host: ozones3_datanode_3.ozones3_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:CLOSED] datanode_1 | 2019-10-15 03:14:55,421 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_2 | 2019-10-15 03:14:55,246 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:56,353 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 20 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13 not found datanode_1 | 2019-10-15 03:14:55,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:56,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,369 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 11 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:132) datanode_1 | 2019-10-15 03:14:55,421 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 11 for changeToCandidate datanode_3 | 2019-10-15 03:14:56,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:322) datanode_3 | 2019-10-15 03:14:56,353 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:55,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:107) datanode_3 | 2019-10-15 03:14:56,463 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 19 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 13 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,370 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:427) datanode_3 | 2019-10-15 03:14:56,463 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,389 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:14:56,463 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:55,389 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,433 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,435 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection9: begin an election at term 12 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:56,463 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,470 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection9: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t12, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t12] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t12, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,389 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 12 for changeToCandidate scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(SCMPipelineManager.java:413) datanode_3 | 2019-10-15 03:14:56,542 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 20 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,470 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 12 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:55,389 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:56,543 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,473 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:55,413 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection9: begin an election at term 13 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,473 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:14:55,482 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_3 | 2019-10-15 03:14:56,543 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.lambda$finalizeAndDestroyPipeline$0(SCMPipelineManager.java:322) datanode_2 | 2019-10-15 03:14:55,440 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection9: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t13, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t13] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t13, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:56,543 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 12 for changeToCandidate datanode_2 | 2019-10-15 03:14:55,441 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 13 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:56,555 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 21 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:55,441 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection scm_1 | at org.apache.hadoop.hdds.utils.Scheduler.lambda$schedule$1(Scheduler.java:68) datanode_3 | 2019-10-15 03:14:56,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,497 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection10: begin an election at term 13 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,441 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:56,555 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,528 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection10: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t13, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t13] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t13, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:14:56,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,454 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 12 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:14:55,528 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 13 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:56,735 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 22 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:56,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,454 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) datanode_1 | 2019-10-15 03:14:55,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:56,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:14:55,540 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 14 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:56,736 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,516 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 13 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c scm_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:14:55,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:56,790 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 23 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,516 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:14:55,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:56,791 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,516 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:19:44,319 INFO container.ReplicationManager: Starting Replication Monitor Thread. datanode_1 | 2019-10-15 03:14:55,542 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:56,791 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted scm_1 | 2019-10-15 03:19:44,332 INFO container.ReplicationManager: Replication Monitor Thread took 10 milliseconds for processing 8 containers. datanode_1 | 2019-10-15 03:14:55,584 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 15 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:56,791 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,516 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState scm_1 | 2019-10-15 03:19:54,223 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 3 blocks datanode_1 | 2019-10-15 03:14:55,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:56,798 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 21 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:55,518 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:74ms scm_1 | 2019-10-15 03:19:54,224 INFO block.BlockManagerImpl: Deleting blocks conID: 6 locID: 102964236597198991 bcsId: 0,conID: 8 locID: 102964236596215949 bcsId: 0,conID: 8 locID: 102964236596215950 bcsId: 0 datanode_1 | 2019-10-15 03:14:55,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:56,798 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,519 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState scm_1 | 2019-10-15 03:19:54,225 INFO block.BlockManagerImpl: Deleting blocks conID: 6 locID: 102964235812339851 bcsId: 0 datanode_1 | 2019-10-15 03:14:55,585 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:56,798 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,519 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 13 for changeToCandidate scm_1 | 2019-10-15 03:19:54,225 INFO block.BlockManagerImpl: Deleting blocks conID: 7 locID: 102964236036800652 bcsId: 0 datanode_1 | 2019-10-15 03:14:55,688 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_3 | 2019-10-15 03:14:56,798 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,519 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:55,689 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:56,993 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 22 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:55,525 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection10: begin an election at term 14 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,689 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 13 for changeToCandidate scm_1 | 2019-10-15 03:20:54,230 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 1 blocks datanode_3 | 2019-10-15 03:14:56,993 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,556 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection10: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t14, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,689 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection scm_1 | 2019-10-15 03:20:54,230 INFO block.BlockManagerImpl: Deleting blocks conID: 6 locID: 102964240927948952 bcsId: 0 datanode_3 | 2019-10-15 03:14:56,994 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,569 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection10: begin an election at term 15 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,703 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection11: begin an election at term 14 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null scm_1 | 2019-10-15 03:21:54,239 INFO server.SCMBlockProtocolServer: SCM is informed by OM to delete 4 blocks datanode_3 | 2019-10-15 03:14:56,994 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,600 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection10: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t15, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t15] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t15, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,719 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 16 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 scm_1 | 2019-10-15 03:21:54,240 INFO block.BlockManagerImpl: Deleting blocks conID: 6 locID: 102964243739508891 bcsId: 0 datanode_3 | 2019-10-15 03:14:57,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 23 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,719 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState scm_1 | 2019-10-15 03:21:54,240 INFO block.BlockManagerImpl: Deleting blocks conID: 7 locID: 102964243958268060 bcsId: 0 datanode_3 | 2019-10-15 03:14:57,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,719 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:55,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 15 for DISCOVERED_A_NEW_TERM scm_1 | 2019-10-15 03:21:54,241 INFO block.BlockManagerImpl: Deleting blocks conID: 7 locID: 102964242063491225 bcsId: 0 datanode_3 | 2019-10-15 03:14:57,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,719 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection scm_1 | 2019-10-15 03:21:54,241 INFO block.BlockManagerImpl: Deleting blocks conID: 8 locID: 102964242796773530 bcsId: 0 datanode_3 | 2019-10-15 03:14:57,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 24 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,738 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection11: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t14, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t14] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t14, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 14 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:55,670 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_3 | 2019-10-15 03:14:57,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:55,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,014 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 15 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,014 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:55,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:57,030 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 25 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,685 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection11: begin an election at term 16 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:55,742 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_2 | 2019-10-15 03:14:55,722 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 14 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:55,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,743 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 14 for changeToCandidate datanode_2 | 2019-10-15 03:14:55,722 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:55,726 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection11: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t16, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t16] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t16, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,030 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,747 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection12: begin an election at term 15 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,074 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 26 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,784 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection12: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t15, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t15] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t15, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,726 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 16 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:57,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,785 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 15 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:55,726 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:57,074 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:55,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:57,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 27 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,835 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_3 | 2019-10-15 03:14:57,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 15 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 15 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:55,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:57,241 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 24 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:55,766 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:57,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,866 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 16 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:57,242 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,850 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection13: begin an election at term 16 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,242 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:57,287 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 28 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:55,866 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,875 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection13: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t16, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t16] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t16, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,287 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:55,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 16 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:57,287 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,881 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_1 | 2019-10-15 03:14:55,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:57,287 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:55,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:57,303 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 25 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:55,881 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 16 for changeToCandidate datanode_1 | 2019-10-15 03:14:55,907 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 17 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:57,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:57,303 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,885 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection12: begin an election at term 17 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:55,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection12: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t17, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t17] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t17, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,377 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 29 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,907 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:55,914 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 17 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:57,377 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,930 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_2 | 2019-10-15 03:14:55,914 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:57,377 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:55,914 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,377 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,931 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 16 for changeToCandidate datanode_2 | 2019-10-15 03:14:55,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 17 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 26 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:55,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:55,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,936 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection14: begin an election at term 17 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 30 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:55,980 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection14: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t17, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t17] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t17, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:55,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:55,980 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 17 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:56,076 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:55,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:56,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:55,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 17 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,114 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 18 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:57,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:57,554 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 31 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:56,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,091 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection13: begin an election at term 18 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,554 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,115 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,132 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection13: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t18, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t18] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t18, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,555 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,167 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_2 | 2019-10-15 03:14:56,134 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 18 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:57,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,134 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:57,574 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 27 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,167 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 17 for changeToCandidate datanode_2 | 2019-10-15 03:14:56,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,574 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:56,149 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_3 | 2019-10-15 03:14:57,574 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection15: begin an election at term 18 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,574 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 19 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,150 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 18 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,691 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 28 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:57,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,185 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,164 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection14: begin an election at term 19 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,691 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,204 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 18 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:57,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,226 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection15: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t18, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t18] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t18, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,715 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 29 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,226 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 18 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:56,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,715 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,204 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:57,715 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,226 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:56,226 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection14: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t19, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t19] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t19, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,226 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,226 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 19 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:56,348 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 20 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:57,715 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:56,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:56,348 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:57,747 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 32 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,314 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_1 | 2019-10-15 03:14:56,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:57,747 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,314 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,427 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_3 | 2019-10-15 03:14:57,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:56,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 19 for changeToCandidate datanode_1 | 2019-10-15 03:14:56,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,748 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:56,427 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 18 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,765 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 33 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,329 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection15: begin an election at term 20 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:57,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,378 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection15: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t20, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t20] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t20, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:57,765 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,379 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 20 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:56,442 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection16: begin an election at term 19 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:57,866 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 30 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,493 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection16: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t19, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t19] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t19, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,866 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,493 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 19 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:56,466 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 19 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:57,866 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,493 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:56,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,866 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,493 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:57,971 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 31 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,533 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_2 | 2019-10-15 03:14:56,466 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:57,971 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,533 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,522 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_3 | 2019-10-15 03:14:57,971 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,534 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 19 for changeToCandidate datanode_3 | 2019-10-15 03:14:57,971 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:57,972 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 34 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,522 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,538 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection17: begin an election at term 20 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:57,972 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,523 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 20 for changeToCandidate datanode_1 | 2019-10-15 03:14:56,556 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 21 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:57,972 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:56,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:57,972 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:56,537 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection16: begin an election at term 21 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,556 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,116 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 32 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:56,558 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 20 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,573 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection17: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t20, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t20] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t20, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,573 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 20 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:56,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:56,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,558 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:58,116 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,732 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 22 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,558 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 33 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,558 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:58,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,573 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection16: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t21, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t21] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t21, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:58,182 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,573 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 21 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:56,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,198 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 35 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:56,733 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,198 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:56,754 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_3 | 2019-10-15 03:14:58,198 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,698 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_1 | 2019-10-15 03:14:56,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,198 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:56,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,754 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 20 for changeToCandidate datanode_3 | 2019-10-15 03:14:58,218 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 36 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 21 for changeToCandidate datanode_1 | 2019-10-15 03:14:56,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:58,218 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:56,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 23 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:58,218 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:56,715 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection17: begin an election at term 22 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,218 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,741 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection17: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t22, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:56,790 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,334 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 34 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:56,772 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection17: begin an election at term 23 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,798 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 21 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:58,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,794 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection18: begin an election at term 21 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,334 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,814 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection18: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t21, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t21] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t21, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:58,395 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 37 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:56,814 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 21 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:56,799 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,395 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,814 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:56,814 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection17: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t23, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t23] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t23, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,395 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:56,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:56,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 23 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:58,395 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:56,962 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:146ms datanode_2 | 2019-10-15 03:14:56,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:56,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,430 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 38 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:56,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:56,963 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 21 for changeToCandidate datanode_3 | 2019-10-15 03:14:58,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:56,992 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 22 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:56,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:58,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:56,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:56,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection19: begin an election at term 22 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,430 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:56,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:56,996 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection19: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t22, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:56,993 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,466 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 35 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,008 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection19: begin an election at term 23 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,003 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_1 | 2019-10-15 03:14:57,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 24 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,466 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,003 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 23 for changeToCandidate datanode_3 | 2019-10-15 03:14:58,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:58,650 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 39 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:57,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,008 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection18: begin an election at term 24 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,015 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,011 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection18: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t24, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,028 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection19: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t23, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t23] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t23, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,650 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,014 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 23 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:58,687 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 40 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,687 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 36 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,028 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 23 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,029 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:57,015 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,026 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection18: begin an election at term 25 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:58,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,030 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 25 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:58,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,045 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection18: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t25, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t25] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t25, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,031 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,046 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 25 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:57,031 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,046 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:57,031 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,716 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 41 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,046 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:58,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,056 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_3 | 2019-10-15 03:14:58,716 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:14:58,734 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 42 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,057 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 25 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 26 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:58,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:57,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,734 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,071 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection19: begin an election at term 26 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection19: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t26, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t26] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t26, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,075 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,846 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 43 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,143 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 27 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,090 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 26 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:58,846 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,846 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:57,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:58,846 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,143 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:58,901 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 37 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,124 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_1 | 2019-10-15 03:14:57,223 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:192ms datanode_3 | 2019-10-15 03:14:58,901 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:58,901 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 26 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,223 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 23 for changeToCandidate datanode_3 | 2019-10-15 03:14:58,901 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:57,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:14:59,034 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 44 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,139 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection20: begin an election at term 27 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,236 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection20: begin an election at term 24 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,151 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection20: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t27, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t27] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t27, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,258 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection20: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t24, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t24] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t24, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,034 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,151 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 27 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:57,259 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 24 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:59,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,151 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:57,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:59,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 38 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,151 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:59,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 45 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 24 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:59,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,287 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 28 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:59,123 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,241 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:59,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,269 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | 2019-10-15 03:14:59,123 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,287 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,269 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,269 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 27 for changeToCandidate datanode_3 | 2019-10-15 03:14:59,224 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 46 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:59,225 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,225 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,269 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:59,225 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,283 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection21: begin an election at term 28 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,239 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 39 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,290 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | 2019-10-15 03:14:57,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection21: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t28, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t28] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t28, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,290 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 24 for changeToCandidate datanode_3 | 2019-10-15 03:14:59,240 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:57,302 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 28 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:57,300 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection21: begin an election at term 25 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 40 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 47 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 25 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,319 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection21: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t25, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t25] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t25, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,319 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 25 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,304 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:59,357 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,349 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:44ms datanode_1 | 2019-10-15 03:14:57,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:59,357 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,379 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 29 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:59,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 28 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,380 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:59ms datanode_3 | 2019-10-15 03:14:59,510 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 41 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:59,510 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,373 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection22: begin an election at term 29 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,510 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,386 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection22: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t29, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,510 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,380 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:59,582 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 48 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,380 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 25 for changeToCandidate datanode_2 | 2019-10-15 03:14:57,393 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection22: begin an election at term 30 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,582 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 26 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:14:59,582 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:57,398 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,582 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,398 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,393 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection22: begin an election at term 26 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,702 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 42 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,398 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,396 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 30 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:59,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,412 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection22: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t30, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t30] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t30, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,412 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 30 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:59,703 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:57,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:14:59,779 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 49 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,396 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:14:59,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,513 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_1 | 2019-10-15 03:14:57,414 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection22: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t26, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t26] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t26, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,414 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 26 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:59,779 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,514 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 30 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,415 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:14:59,860 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 43 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,415 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:14:59,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,554 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 31 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:14:59,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,555 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,550 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection23: begin an election at term 31 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,861 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,555 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,572 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection23: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t31, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t31] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t31, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:14:59,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 50 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,555 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,572 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 31 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:14:59,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,572 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:57,559 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | 2019-10-15 03:14:59,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,572 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,559 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:14:59,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,576 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 27 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,559 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 26 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,007 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 51 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,007 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,560 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:57,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,007 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,570 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection23: begin an election at term 27 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,577 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:00,007 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,602 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection23: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t27, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t27] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t27, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,691 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 28 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:00,064 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 44 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 27 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:57,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:00,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,691 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,671 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_3 | 2019-10-15 03:15:00,064 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,715 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 29 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,219 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 52 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 27 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:57,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:00,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,715 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,687 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection24: begin an election at term 28 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,219 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,730 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_1 | 2019-10-15 03:14:57,697 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection24: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t28, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,249 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 45 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection24: begin an election at term 29 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,730 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 31 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,731 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection24: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t29, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t29] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t29, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 29 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:00,250 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:57,744 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection24: begin an election at term 32 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:57,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,755 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection24: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t32, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t32] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t32, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 53 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,747 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 32 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,755 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 32 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:00,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,748 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:00,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,748 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,748 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:00,398 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 46 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,765 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 33 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,757 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_3 | 2019-10-15 03:15:00,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:00,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,765 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,757 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 32 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,430 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 54 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,848 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_2 | 2019-10-15 03:14:57,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:00,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,762 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection25: begin an election at term 33 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,430 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:57,784 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection25: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t33, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t33] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t33, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,848 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 29 for changeToCandidate datanode_2 | 2019-10-15 03:14:57,784 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 33 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:00,449 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 55 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:57,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:00,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,862 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection25: begin an election at term 30 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,449 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,882 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection25: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t30, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t30] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t30, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,865 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 30 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:00,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,532 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 56 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,882 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 30 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,882 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:57,866 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:00,532 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,882 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,949 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:14:57,953 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_3 | 2019-10-15 03:15:00,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:57,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,533 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 47 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:57,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 33 for changeToCandidate datanode_1 | 2019-10-15 03:14:57,954 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 30 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:57,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:57,968 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection26: begin an election at term 31 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,968 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection26: begin an election at term 34 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,972 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 34 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:57,971 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 31 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:00,534 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:57,971 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,688 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 57 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:57,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,971 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:14:57,971 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,972 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:00,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:57,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection26: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t31, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t31] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t31, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:57,988 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection26: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t34, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t34] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t34, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:57,988 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 31 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:57,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 34 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:00,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:57,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:57,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:00,733 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 48 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:57,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:57,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,097 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:14:58,116 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 32 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:00,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,734 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 31 for changeToCandidate datanode_2 | 2019-10-15 03:14:58,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,753 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 49 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:58,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:58,116 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:00,753 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,112 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection27: begin an election at term 32 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:58,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 33 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:00,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,130 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection27: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t32, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t32] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t32, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:58,182 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,754 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,130 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 32 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:00,898 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 58 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:58,130 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:58,182 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:00,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,130 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:58,182 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:00,899 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,155 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_2 | 2019-10-15 03:14:58,188 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_3 | 2019-10-15 03:15:00,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:00,924 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 50 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:58,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 32 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,924 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:00,924 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,175 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection28: begin an election at term 33 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:58,189 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 34 for changeToCandidate datanode_3 | 2019-10-15 03:15:00,924 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,197 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection28: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t33, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t33] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t33, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:58,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:00,963 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 59 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:58,197 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 33 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:58,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection27: begin an election at term 35 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,197 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:58,211 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection27: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t35, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,198 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:58,215 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection27: begin an election at term 36 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:00,963 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,198 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 35 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,233 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection27: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t36, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t36] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t36, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 36 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:01,020 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 51 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:58,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,020 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:58,201 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:01,021 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,217 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 36 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,335 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 34 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:01,021 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 60 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:58,218 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:58,335 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,218 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,375 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_3 | 2019-10-15 03:15:01,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,317 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | 2019-10-15 03:14:58,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,208 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 36 for changeToCandidate datanode_1 | 2019-10-15 03:14:58,318 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 33 for changeToCandidate datanode_3 | 2019-10-15 03:15:01,233 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 52 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:58,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:01,233 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,390 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection28: begin an election at term 37 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,331 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection29: begin an election at term 34 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,233 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,411 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection28: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t37, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t37] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t37, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,233 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,349 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection29: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t34, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t34] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t34, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:58,411 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 37 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:01,299 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 53 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:58,349 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 34 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:01,299 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:01,299 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,299 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,413 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_1 | 2019-10-15 03:14:58,395 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 37 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:01,326 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 61 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,327 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,414 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 37 for changeToCandidate datanode_1 | 2019-10-15 03:14:58,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,327 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:58,395 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:01,327 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,427 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection29: begin an election at term 38 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 38 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:01,395 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 62 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection29: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t38, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t38] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t38, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,395 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,446 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 38 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:58,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,395 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:58,430 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:01,395 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,449 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:15:01,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 54 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,466 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 35 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:58,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,449 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 34 for changeToCandidate datanode_3 | 2019-10-15 03:15:01,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:01,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,467 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,463 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection30: begin an election at term 35 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,463 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 63 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection30: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t35, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t35] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t35, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,463 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,634 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_1 | 2019-10-15 03:14:58,482 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 35 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:01,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:01,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,634 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 38 for changeToCandidate datanode_1 | 2019-10-15 03:14:58,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,608 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 55 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:58,652 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 39 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:01,608 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,646 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection30: begin an election at term 39 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,652 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,608 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,669 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection30: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t39, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t39] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t39, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,609 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,669 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 39 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:58,653 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:01,679 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 56 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:58,653 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,680 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,670 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_3 | 2019-10-15 03:15:01,682 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,673 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_1 | 2019-10-15 03:14:58,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,682 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 35 for changeToCandidate datanode_3 | 2019-10-15 03:15:01,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 64 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,674 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 39 for changeToCandidate datanode_1 | 2019-10-15 03:14:58,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:01,708 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:58,684 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection31: begin an election at term 36 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,708 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,684 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: begin an election at term 40 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 40 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:01,708 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 57 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 36 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,687 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:01,708 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,709 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,715 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection31: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t36, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t36] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t36, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,710 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,694 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t40, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,715 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 36 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:01,710 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,712 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: begin an election at term 41 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:01,806 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 65 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,719 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t41, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,730 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: begin an election at term 42 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,718 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 41 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:01,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection31: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t42, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t42] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t42, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:58,718 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,718 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:58,750 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 42 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:58,718 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,750 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:01,806 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,734 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 42 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,750 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:01,843 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 58 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,827 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_1 | 2019-10-15 03:14:58,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:01,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:01,843 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,734 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 42 for changeToCandidate datanode_3 | 2019-10-15 03:15:01,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,846 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 43 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:58,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:01,986 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 66 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:58,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:58,843 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection32: begin an election at term 43 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:58,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection32: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t43, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t43] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t43, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:01,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:58,846 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,862 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 43 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:58,867 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | 2019-10-15 03:15:01,986 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,862 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:58,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,014 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 59 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,862 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:02,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,867 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 36 for changeToCandidate datanode_2 | 2019-10-15 03:14:58,901 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 37 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:02,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:58,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,868 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:02,014 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:58,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:58,898 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection32: begin an election at term 37 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 60 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:58,904 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:58,916 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection32: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t37, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t37] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t37, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,016 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_1 | 2019-10-15 03:14:58,916 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 37 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:58,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:59,017 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 43 for changeToCandidate datanode_1 | 2019-10-15 03:14:58,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:02,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 44 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,031 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection33: begin an election at term 44 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,211 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 67 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,050 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection33: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t44, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t44] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t44, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,050 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 44 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,035 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,212 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,105 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_2 | 2019-10-15 03:14:59,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:02,312 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 68 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,115 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_1 | 2019-10-15 03:14:59,105 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 37 for changeToCandidate datanode_2 | 2019-10-15 03:14:59,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,312 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,115 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 44 for changeToCandidate datanode_3 | 2019-10-15 03:15:02,312 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:02,312 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection33: begin an election at term 38 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:59,122 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 45 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:02,340 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 61 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:59,119 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection34: begin an election at term 45 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:59,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 38 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:02,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:02,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,122 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:02,340 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,136 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection33: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t38, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t38] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t38, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 62 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:59,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,136 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 38 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,136 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,136 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:02,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,223 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:15:02,568 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 63 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:59,136 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection34: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t45, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t45] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t45, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:59,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,136 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 45 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:14:59,223 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 38 for changeToCandidate datanode_3 | 2019-10-15 03:15:02,568 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 69 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:59,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,224 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 46 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,568 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:02,569 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,200 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_3 | 2019-10-15 03:15:02,569 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,634 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 64 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:14:59,201 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 45 for changeToCandidate datanode_3 | 2019-10-15 03:15:02,634 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:59,224 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:02,634 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,221 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection35: begin an election at term 46 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,634 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,237 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection34: begin an election at term 39 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,238 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection35: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t46, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t46] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t46, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,679 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 70 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,255 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection34: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t39, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t39] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t39, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,239 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 46 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,679 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,255 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 39 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:59,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,680 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,255 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:59,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:02,680 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 39 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:02,868 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 71 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,339 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_2 | 2019-10-15 03:14:59,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,339 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 39 for changeToCandidate datanode_2 | 2019-10-15 03:14:59,240 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:59,339 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:15:02,868 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 65 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,353 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection35: begin an election at term 40 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,356 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 47 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 46 for changeToCandidate datanode_3 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:59,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,353 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection36: begin an election at term 47 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:59,356 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,356 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 40 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,371 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection35: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t40, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t40] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t40, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,869 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,371 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 40 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,869 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,887 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 72 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,356 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:02,887 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,371 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection36: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t47, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t47] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t47, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,887 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,491 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | 2019-10-15 03:14:59,372 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 47 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,888 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:14:59,491 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 40 for changeToCandidate datanode_3 | 2019-10-15 03:15:02,929 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 73 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:02,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,507 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection36: begin an election at term 41 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,510 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 41 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,518 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection36: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t41, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t41] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t41, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:02,930 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,518 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 41 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:02,958 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 74 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:02,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,510 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:02,959 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,565 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | 2019-10-15 03:15:02,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,565 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,582 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 48 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,566 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 47 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 75 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:14:59,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:03,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,579 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection37: begin an election at term 48 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,013 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,582 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,590 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection37: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t48, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t48] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t48, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,690 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:03,039 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 66 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,039 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,690 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 41 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,039 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:59,590 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 48 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:03,039 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection37: begin an election at term 42 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 67 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,710 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection37: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t42, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t42] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t42, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 42 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:03,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,711 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 42 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:59,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,184 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 76 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,703 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:14:59,779 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 49 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:03,184 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,763 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_1 | 2019-10-15 03:14:59,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,779 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:03,301 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 77 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,763 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 48 for changeToCandidate datanode_1 | 2019-10-15 03:14:59,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | 2019-10-15 03:15:03,301 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:14:59,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,776 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection38: begin an election at term 49 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:14:59,834 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 42 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,301 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:14:59,789 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection38: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t49, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t49] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t49, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,301 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:14:59,790 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 49 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:03,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 68 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,856 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection38: begin an election at term 43 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,869 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection38: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t43, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t43] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t43, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:14:59,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 43 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:03,362 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,869 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 43 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:14:59,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,386 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 69 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:14:59,862 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,386 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,862 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:03,386 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:14:59,927 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 50 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,909 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_3 | 2019-10-15 03:15:03,386 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:14:59,927 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 78 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:14:59,927 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:14:59,910 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 49 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 70 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:14:59,927 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:00,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 51 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,923 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection39: begin an election at term 50 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,950 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection39: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t50, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t50] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t50, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,950 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 50 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,007 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,950 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,047 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:14:59,951 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,989 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_1 | 2019-10-15 03:15:00,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 43 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:14:59,989 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:03,499 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 79 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:14:59,989 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 50 for changeToCandidate datanode_1 | 2019-10-15 03:15:00,061 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection39: begin an election at term 44 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:14:59,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:00,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection39: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t44, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t44] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t44, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,500 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection40: begin an election at term 51 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 44 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:03,500 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,022 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection40: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t51, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t51] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t51, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,600 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 71 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,022 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 51 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:00,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,600 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,600 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,022 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 52 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:03,600 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,729 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 80 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,729 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,064 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 44 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,729 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,220 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,729 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,231 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_2 | 2019-10-15 03:15:00,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,779 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 72 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,065 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,231 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 44 for changeToCandidate datanode_2 | 2019-10-15 03:15:00,202 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:178ms datanode_3 | 2019-10-15 03:15:03,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:03,779 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:03,964 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 81 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,202 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 51 for changeToCandidate datanode_3 | 2019-10-15 03:15:03,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection40: begin an election at term 45 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:03,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection40: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t45, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t45] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t45, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,216 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection41: begin an election at term 52 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,964 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 45 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:00,227 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection41: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t52, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t52] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t52, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:03,979 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 73 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,227 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 52 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:00,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:03,979 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,382 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | 2019-10-15 03:15:00,249 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 45 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,382 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 45 for changeToCandidate datanode_3 | 2019-10-15 03:15:04,053 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 82 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:00,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:04,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,250 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,393 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection41: begin an election at term 46 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,378 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_1 | 2019-10-15 03:15:00,399 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 53 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:04,053 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,399 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,185 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 74 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,378 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 52 for changeToCandidate datanode_1 | 2019-10-15 03:15:00,399 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:04,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,399 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,393 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: begin an election at term 53 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,414 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection41: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t46, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t46] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t46, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,185 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 46 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,414 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 46 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,186 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 83 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,414 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,415 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:00,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 54 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,187 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,430 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,398 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,448 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 55 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:04,381 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 84 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,398 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:04,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,449 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,413 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t53, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,381 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,497 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_2 | 2019-10-15 03:15:00,427 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: begin an election at term 54 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 75 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,440 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t54, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,444 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: begin an election at term 55 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,478 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection42: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t55, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t55] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t55, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,478 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 55 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:00,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,497 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 46 for changeToCandidate datanode_1 | 2019-10-15 03:15:00,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:00,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,529 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection42: begin an election at term 47 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,531 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 56 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:04,489 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 76 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:04,489 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,508 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_1 | 2019-10-15 03:15:00,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:04,490 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,509 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,532 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:04,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,509 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 55 for changeToCandidate datanode_1 | 2019-10-15 03:15:00,549 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection42: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t47, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t47] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t47, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,558 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 77 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,509 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:00,550 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 47 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,558 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,529 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection43: begin an election at term 56 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:04,558 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,533 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 47 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:04,558 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,533 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,577 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 85 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,686 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 57 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,533 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:04,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,533 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:04,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,549 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection43: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t56, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t56] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t56, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,577 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,687 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,549 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 56 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,674 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 78 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,711 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_2 | 2019-10-15 03:15:00,549 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:04,675 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,549 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:00,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,675 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,667 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_1 | 2019-10-15 03:15:00,711 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 47 for changeToCandidate datanode_2 | 2019-10-15 03:15:00,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,675 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:00,668 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 56 for changeToCandidate datanode_3 | 2019-10-15 03:15:04,698 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 86 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,726 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection43: begin an election at term 48 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:04,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,728 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection43: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t48, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,683 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection44: begin an election at term 57 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,713 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection44: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t57, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t57] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t57, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,750 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection43: begin an election at term 49 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,699 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 57 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:00,769 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection43: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t49, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t49] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t49, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,821 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 79 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,769 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 49 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:00,769 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:04,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,729 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 48 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:04,821 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,769 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,729 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,873 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 87 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,898 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 58 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,730 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:04,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:04,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,753 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 49 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,899 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,873 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:00,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:04,939 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 80 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,754 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,909 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_3 | 2019-10-15 03:15:04,939 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,882 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_1 | 2019-10-15 03:15:00,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,909 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 49 for changeToCandidate datanode_2 | 2019-10-15 03:15:00,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 57 for changeToCandidate datanode_1 | 2019-10-15 03:15:00,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:04,940 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:00,922 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection44: begin an election at term 50 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,895 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection45: begin an election at term 58 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 88 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,931 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection44: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t50, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t50] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t50, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:00,906 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection45: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t58, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t58] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t58, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:00,932 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 50 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,932 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,932 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:00,962 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 59 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:00,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 81 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:00,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,906 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 58 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:04,980 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,962 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:00,994 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:00,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,924 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 50 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:00,995 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 50 for changeToCandidate datanode_3 | 2019-10-15 03:15:04,980 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:00,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:00,924 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,060 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 82 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:00,924 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:05,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,924 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:05,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:00,954 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_1 | 2019-10-15 03:15:01,017 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection45: begin an election at term 51 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,060 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,037 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection45: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t51, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t51] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t51, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,085 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 89 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,955 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 58 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,037 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 51 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:00,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:05,085 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:00,960 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection46: begin an election at term 59 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:05,085 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,208 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 60 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:00,994 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection46: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t59, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t59] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t59, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,085 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,208 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:00,994 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 59 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 83 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:00,994 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 90 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,209 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:00,994 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,216 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:15:01,021 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 51 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,175 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,216 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 51 for changeToCandidate datanode_2 | 2019-10-15 03:15:01,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,021 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,230 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection46: begin an election at term 52 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,191 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | 2019-10-15 03:15:05,175 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,249 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection46: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t52, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t52] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t52, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,373 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 84 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,249 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 52 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:01,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 59 for changeToCandidate datanode_3 | 2019-10-15 03:15:05,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:01,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:05,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,205 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection47: begin an election at term 60 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,373 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,282 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_2 | 2019-10-15 03:15:01,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection47: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t60, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t60] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t60, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,374 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 91 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,217 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 60 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:01,282 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 52 for changeToCandidate datanode_3 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:01,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:05,374 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection47: begin an election at term 53 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,233 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 52 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection47: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t53, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t53] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t53, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:05,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 85 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,324 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 53 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:01,233 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:05,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:01,294 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_3 | 2019-10-15 03:15:05,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 92 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,295 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 60 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:05,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:01,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 61 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:05,708 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,299 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 53 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,708 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:05,798 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 93 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,326 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:05,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,300 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:05,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,394 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 62 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:05,799 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,321 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection48: begin an election at term 61 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,394 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,869 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 86 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,342 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection48: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t61, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t61] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t61, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,394 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:05,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,342 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 61 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:01,394 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:05,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,402 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_3 | 2019-10-15 03:15:05,870 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,342 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:05,990 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 94 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,402 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:05,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,402 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 53 for changeToCandidate datanode_2 | 2019-10-15 03:15:01,377 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_3 | 2019-10-15 03:15:05,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,403 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:05,990 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,377 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 61 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,419 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection48: begin an election at term 54 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,102 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 87 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:01,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection48: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t54, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t54] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t54, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,391 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection49: begin an election at term 62 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 54 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:01,422 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection49: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t62, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t62] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t62, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,103 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:01,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 54 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:06,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,463 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 63 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 88 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,422 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,463 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,127 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 62 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:01,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:01,589 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:15:01,423 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 95 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,437 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_1 | 2019-10-15 03:15:01,589 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 54 for changeToCandidate datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:06,127 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,437 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 62 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,604 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection49: begin an election at term 55 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:01,625 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection49: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t55, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t55] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t55, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,283 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 96 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,460 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection50: begin an election at term 63 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,625 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 55 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:06,283 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,495 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection50: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t63, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t63] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t63, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:06,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:06,284 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,496 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 63 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:01,662 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | 2019-10-15 03:15:06,284 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 89 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:01,662 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 55 for changeToCandidate datanode_3 | 2019-10-15 03:15:06,284 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:06,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,608 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 55 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,676 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection50: begin an election at term 56 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,608 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,682 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection50: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t56, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,609 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,480 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 97 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,609 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,703 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection50: begin an election at term 57 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,677 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_1 | 2019-10-15 03:15:01,711 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 64 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:06,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,480 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,677 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 63 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:01,711 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,526 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 90 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,681 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 56 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:06,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,733 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection50: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t57, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t57] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t57, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,527 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,733 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 57 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:01,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:06,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:01,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,686 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 98 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:06,686 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,703 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection51: begin an election at term 64 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,686 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,706 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 57 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,805 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 65 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:06,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,760 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 91 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:06,760 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,707 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,761 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,805 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,734 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection51: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t64, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t64] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t64, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:06,761 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,826 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | 2019-10-15 03:15:01,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 64 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:06,820 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 99 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,820 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:01,826 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 57 for changeToCandidate datanode_2 | 2019-10-15 03:15:01,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:01,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:01,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:06,820 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,840 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection51: begin an election at term 58 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,789 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_3 | 2019-10-15 03:15:06,820 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:01,858 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection51: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t58, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t58] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t58, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,845 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 92 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:01,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 64 for changeToCandidate datanode_1 | 2019-10-15 03:15:01,858 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 58 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:06,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:01,858 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:06,846 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection52: begin an election at term 65 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,859 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,822 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection52: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t65, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t65] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t65, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:01,979 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_3 | 2019-10-15 03:15:06,846 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,822 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 65 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:01,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,923 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 100 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:01,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:01,979 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 58 for changeToCandidate datanode_3 | 2019-10-15 03:15:06,923 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,823 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:01,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:01,843 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 58 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 66 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:06,923 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,923 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:06,987 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 101 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:01,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,843 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:06,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:01,967 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | 2019-10-15 03:15:06,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:01,967 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:06,987 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:01,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:01,967 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 65 for changeToCandidate datanode_3 | 2019-10-15 03:15:07,030 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 93 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:01,986 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:01,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:07,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,008 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection52: begin an election at term 59 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,029 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection52: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t59, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t59] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t59, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:01,982 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection53: begin an election at term 66 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,030 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,011 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection53: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t66, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t66] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t66, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,029 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 59 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,083 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 94 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,011 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 66 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,083 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,029 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:02,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:07,084 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,029 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:07,084 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,175 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_2 | 2019-10-15 03:15:02,014 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 59 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:07,219 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 102 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,175 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 59 for changeToCandidate datanode_3 | 2019-10-15 03:15:07,219 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:02,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:07,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,189 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection53: begin an election at term 60 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,014 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:07,288 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 95 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,210 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection53: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t60, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t60] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t60, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,194 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 60 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:07,288 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,211 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 60 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,288 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:07,288 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 103 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:07,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,366 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,212 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 67 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:07,477 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 96 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:07,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:07,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,212 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,199 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_3 | 2019-10-15 03:15:07,477 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,571 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 104 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,311 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 68 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:07,571 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:07,571 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,199 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 66 for changeToCandidate datanode_1 | 2019-10-15 03:15:02,312 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:07,571 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:02,316 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:103ms datanode_3 | 2019-10-15 03:15:07,607 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 97 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,208 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection54: begin an election at term 67 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,607 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection54: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t67, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t67] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t67, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,316 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,607 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,316 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 60 for changeToCandidate datanode_3 | 2019-10-15 03:15:07,607 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:02,239 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 67 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 98 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,337 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection54: begin an election at term 61 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:02,366 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection54: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t61, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t61] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t61, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:02,366 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 61 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,294 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_1 | 2019-10-15 03:15:02,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:07,783 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,294 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 105 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,294 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 67 for changeToCandidate datanode_1 | 2019-10-15 03:15:02,472 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | 2019-10-15 03:15:02,294 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,308 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection55: begin an election at term 68 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,472 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 61 for changeToCandidate datanode_2 | 2019-10-15 03:15:02,339 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection55: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t68, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t68] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t68, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,783 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:02,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 68 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:07,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 106 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,485 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection55: begin an election at term 62 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:07,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,516 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection55: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t62, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t62] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t62, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:07,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 62 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:02,340 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 61 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:07,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:02,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,921 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 107 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,517 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:07,921 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,542 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_2 | 2019-10-15 03:15:02,340 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:07,921 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,487 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 62 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:07,921 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,542 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 62 for changeToCandidate datanode_3 | 2019-10-15 03:15:07,979 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 99 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:02,565 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection56: begin an election at term 63 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:07,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:07,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:07,979 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 108 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,568 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 69 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:08,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,568 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,006 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,582 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection56: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t63, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t63] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t63, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,104 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 100 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,488 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,583 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 63 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:08,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:08,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,540 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_3 | 2019-10-15 03:15:08,104 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,229 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 109 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,540 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 68 for changeToCandidate datanode_1 | 2019-10-15 03:15:02,608 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_2 | 2019-10-15 03:15:02,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:08,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,608 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,566 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection56: begin an election at term 69 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,609 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 63 for changeToCandidate datanode_2 | 2019-10-15 03:15:02,568 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 63 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:08,230 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,609 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:02,568 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,275 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 110 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection57: begin an election at term 64 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:08,275 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,660 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection57: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t64, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t64] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t64, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,569 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,275 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,660 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 64 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:08,275 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,583 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection56: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t69, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t69] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t69, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:08,300 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 101 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 69 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:02,679 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 70 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:08,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:02,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,300 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:02,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 102 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,634 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 64 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,679 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,838 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_3 | 2019-10-15 03:15:08,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:08,439 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,634 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,838 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,478 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 111 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,665 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_3 | 2019-10-15 03:15:08,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,838 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 64 for changeToCandidate datanode_3 | 2019-10-15 03:15:08,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,666 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 69 for changeToCandidate datanode_1 | 2019-10-15 03:15:02,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:08,478 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,666 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:02,866 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection58: begin an election at term 65 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,676 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection57: begin an election at term 70 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,570 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 112 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:02,868 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 71 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,721 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection57: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t70, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t70] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t70, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,570 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 70 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:08,570 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:08,571 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:08,632 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 103 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,869 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,839 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | 2019-10-15 03:15:08,632 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,886 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection58: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t65, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t65] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t65, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:02,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,633 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,886 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 65 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:02,839 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 70 for changeToCandidate datanode_1 | 2019-10-15 03:15:02,887 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:08,633 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:02,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,703 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 104 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:02,866 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: begin an election at term 71 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,888 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 72 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:08,703 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 113 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,868 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 65 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:02,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:02,888 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:02,929 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 73 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:02,868 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,704 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,870 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t71, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,885 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: begin an election at term 72 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,704 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:02,923 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t72, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,930 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,927 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: begin an election at term 73 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,787 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 105 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:02,957 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 74 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:08,787 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:02,938 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t73, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:02,955 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: begin an election at term 74 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:02,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:08,788 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:02,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:02,981 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t74, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,806 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 114 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:03,009 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 75 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,006 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: begin an election at term 75 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,037 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection58: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t75, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t75] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t75, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,806 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:03,037 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 75 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:08,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:08,957 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 106 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:03,037 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,009 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:08,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,037 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,025 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | 2019-10-15 03:15:08,957 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,038 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 66 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:08,985 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 115 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:03,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 65 for changeToCandidate datanode_2 | 2019-10-15 03:15:03,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:08,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,038 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,036 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection59: begin an election at term 66 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 67 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,055 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection59: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t66, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t66] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t66, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:08,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 66 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:08,985 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:09,144 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 116 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,143 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,166 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_1 | 2019-10-15 03:15:03,124 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_3 | 2019-10-15 03:15:09,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,125 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,144 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,166 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 75 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,125 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 66 for changeToCandidate datanode_3 | 2019-10-15 03:15:09,185 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 107 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:03,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,125 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:03,181 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection59: begin an election at term 76 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection60: begin an election at term 67 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,206 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection59: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t76, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t76] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t76, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,185 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,158 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection60: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t67, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t67] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t67, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,207 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 76 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:09,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,158 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 67 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:03,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:09,234 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 117 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,259 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_1 | 2019-10-15 03:15:03,183 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 76 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:09,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,234 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,289 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 118 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,259 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 76 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,183 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,301 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 77 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:09,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,298 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection60: begin an election at term 77 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,289 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:03,319 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection60: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t77, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t77] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t77, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,378 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 108 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,301 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,378 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,319 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 77 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:03,334 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_3 | 2019-10-15 03:15:09,378 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,319 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,378 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,334 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 67 for changeToCandidate datanode_2 | 2019-10-15 03:15:03,319 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:09,499 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 109 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:03,362 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 68 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:09,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,360 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection61: begin an election at term 68 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,363 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,370 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection61: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t68, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,363 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,384 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection61: begin an election at term 69 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,363 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,499 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,386 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 69 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:09,514 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 119 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:09,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,386 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,386 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,412 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection61: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t69, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t69] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t69, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,514 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,386 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,585 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 120 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,447 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_3 | 2019-10-15 03:15:09,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:09,585 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,645 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 110 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:03,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,645 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,447 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 77 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,413 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 69 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:09,645 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,413 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:09,645 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,468 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection61: begin an election at term 78 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:03,413 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 121 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,470 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 70 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,453 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_2 | 2019-10-15 03:15:03,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 111 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:03,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,453 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 69 for changeToCandidate datanode_2 | 2019-10-15 03:15:03,475 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection61: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t78, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:03,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection61: begin an election at term 79 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,468 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection62: begin an election at term 70 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,526 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection61: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t79, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t79] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t79, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:03,471 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 78 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,526 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 79 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:03,527 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,471 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,599 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 71 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:09,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,499 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection62: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t70, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t70] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t70, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,824 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 112 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:03,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 70 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:09,824 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,600 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:09,824 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,700 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,824 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 79 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:09,858 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 122 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,700 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 79 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,499 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,725 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection62: begin an election at term 80 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,858 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,572 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_2 | 2019-10-15 03:15:03,745 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection62: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t80, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t80] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t80, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 113 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,746 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 80 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 123 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:03,572 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 70 for changeToCandidate datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,746 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,746 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,596 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection63: begin an election at term 71 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,779 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 72 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:09,916 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection63: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t71, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t71] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t71, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,615 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 71 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:03,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:09,917 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:03,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:03,935 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_1 | 2019-10-15 03:15:03,729 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 80 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:09,987 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 124 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:03,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,935 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 80 for changeToCandidate datanode_3 | 2019-10-15 03:15:09,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,729 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:03,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:09,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:09,988 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,761 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:144ms datanode_2 | 2019-10-15 03:15:03,961 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection63: begin an election at term 81 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,011 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 125 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:03,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,978 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection63: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t81, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t81] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t81, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,011 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,761 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 71 for changeToCandidate datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 81 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:10,011 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:03,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:10,011 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,777 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection64: begin an election at term 72 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 73 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:10,125 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 126 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,807 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection64: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t72, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t72] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t72, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,125 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,125 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,807 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 72 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:03,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,125 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:03,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:03,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:10,153 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 114 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:03,808 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,035 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_3 | 2019-10-15 03:15:10,153 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,963 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 81 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:10,153 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,035 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 81 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,153 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:10,305 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 127 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,048 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection64: begin an election at term 82 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:03,963 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:10,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,081 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection64: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t82, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t82] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t82, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:03,968 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_3 | 2019-10-15 03:15:10,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,082 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 82 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:03,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,306 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,082 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:03,968 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 72 for changeToCandidate datanode_3 | 2019-10-15 03:15:10,319 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 115 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,082 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:03,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:10,319 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,173 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:91ms, electionTimeout:91ms datanode_1 | 2019-10-15 03:15:03,977 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection65: begin an election at term 73 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,319 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:03,994 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection65: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t73, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t73] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t73, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,319 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,174 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 82 for changeToCandidate datanode_1 | 2019-10-15 03:15:03,994 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 73 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:10,469 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 116 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:03,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:10,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,182 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection65: begin an election at term 83 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:03,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:10,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,186 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 74 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:10,470 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,052 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 82 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,526 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 128 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,052 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,052 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,187 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:10,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,052 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,203 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection65: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t83, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t83] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t83, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,526 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,167 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_2 | 2019-10-15 03:15:04,203 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 83 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:10,587 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 117 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:10,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,167 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 73 for changeToCandidate datanode_2 | 2019-10-15 03:15:04,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,587 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:04,345 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_3 | 2019-10-15 03:15:10,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection66: begin an election at term 74 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:04,345 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,186 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 83 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,345 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 83 for changeToCandidate datanode_3 | 2019-10-15 03:15:10,700 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 129 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,345 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:04,186 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:10,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,187 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:10,700 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,203 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection66: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t74, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t74] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t74, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,367 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection66: begin an election at term 84 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 74 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:10,764 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 118 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 75 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:10,764 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,341 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_2 | 2019-10-15 03:15:04,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,765 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,341 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:10,829 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 119 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,341 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 74 for changeToCandidate datanode_2 | 2019-10-15 03:15:04,424 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection66: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t84, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t84] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t84, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,341 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:10,829 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,425 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 84 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:04,390 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection67: begin an election at term 75 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:10,829 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:04,395 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 84 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:10,829 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,858 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 130 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,489 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 76 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:10,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,397 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,858 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,425 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection67: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t75, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t75] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t75, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:04,489 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:10,958 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 131 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,425 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 75 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:04,554 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | 2019-10-15 03:15:10,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,425 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:04,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:10,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 84 for changeToCandidate datanode_1 | 2019-10-15 03:15:04,478 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_3 | 2019-10-15 03:15:10,959 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:04,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:10,981 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 120 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,558 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 77 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,478 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 75 for changeToCandidate datanode_3 | 2019-10-15 03:15:10,981 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,558 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:10,981 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,558 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:10,981 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,487 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection68: begin an election at term 76 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:04,558 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:11,022 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 132 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,504 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection68: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t76, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t76] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t76, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,022 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,573 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection67: begin an election at term 85 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,505 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 76 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,592 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection67: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t85, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t85] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t85, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:11,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,592 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 85 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,113 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 133 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,592 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:04,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:11,113 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,592 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:11,114 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,674 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 78 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,541 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | 2019-10-15 03:15:11,114 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 121 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,675 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 76 for changeToCandidate datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 134 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,675 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,555 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection69: begin an election at term 77 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_1 | 2019-10-15 03:15:04,574 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection69: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t77, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t77] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t77, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,575 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 77 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,181 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 85 for changeToCandidate datanode_1 | 2019-10-15 03:15:04,575 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:11,181 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:04,575 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:11,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,696 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection68: begin an election at term 86 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 122 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,576 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 85 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,713 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection68: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t86, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t86] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t86, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,577 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 86 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,577 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:11,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,577 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:11,398 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 135 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,658 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:83ms, electionTimeout:83ms datanode_2 | 2019-10-15 03:15:04,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:11,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,821 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 79 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:11,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,659 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 77 for changeToCandidate datanode_2 | 2019-10-15 03:15:04,821 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:04,821 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:11,425 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 123 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,672 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection70: begin an election at term 78 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:04,822 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:11,425 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,857 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_1 | 2019-10-15 03:15:04,697 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection70: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t78, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t78] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t78, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,425 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,857 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,697 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 78 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,425 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:04,857 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 86 for changeToCandidate datanode_3 | 2019-10-15 03:15:11,459 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 136 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:04,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:04,857 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:11,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 86 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,871 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection69: begin an election at term 87 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,459 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,887 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection69: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t87, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t87] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t87, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:11,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,887 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 87 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:04,698 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:11,489 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 137 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:04,804 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_3 | 2019-10-15 03:15:11,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,940 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 80 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 78 for changeToCandidate datanode_3 | 2019-10-15 03:15:11,490 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:11,646 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 124 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:04,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection71: begin an election at term 79 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,646 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,940 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,836 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection71: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t79, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t79] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t79, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,646 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,962 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_1 | 2019-10-15 03:15:04,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 79 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:11,646 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:11,726 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 138 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,962 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 87 for changeToCandidate datanode_1 | 2019-10-15 03:15:04,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:11,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:04,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 87 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:11,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,977 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection70: begin an election at term 88 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,726 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 81 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:11,831 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 139 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,873 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:11,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,923 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:15:11,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,832 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:04,999 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection70: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t88, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t88] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t88, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,923 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 79 for changeToCandidate datanode_3 | 2019-10-15 03:15:11,875 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 140 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:04,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 88 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:04,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:11,875 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:04,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:04,937 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection72: begin an election at term 80 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,875 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:04,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,964 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection72: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t80, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t80] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t80, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:11,875 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,055 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_3 | 2019-10-15 03:15:11,954 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 125 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,964 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 80 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:05,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:11,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:05,056 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 88 for changeToCandidate datanode_3 | 2019-10-15 03:15:11,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:04,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:05,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:11,954 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,973 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_3 | 2019-10-15 03:15:12,035 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 126 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:05,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 82 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,973 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 80 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:04,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:12,035 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,059 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,977 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection73: begin an election at term 81 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,051 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 141 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,082 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection71: begin an election at term 89 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,979 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 88 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:12,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:12,051 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:12,091 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 142 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:12,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,099 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection71: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t89, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t89] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t89, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:04,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,092 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,979 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,099 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 89 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:12,248 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 127 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:04,999 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection73: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t81, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t81] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t81, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,249 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,099 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:04,999 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 81 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:12,249 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,099 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:12,249 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:04,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:05,158 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_3 | 2019-10-15 03:15:12,274 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 143 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:04,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 89 for changeToCandidate datanode_1 | 2019-10-15 03:15:05,031 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_3 | 2019-10-15 03:15:12,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:05,031 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,274 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,172 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection72: begin an election at term 90 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 81 for changeToCandidate datanode_2 | 2019-10-15 03:15:05,175 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 83 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:12,349 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 128 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection74: begin an election at term 82 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,175 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:12,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,083 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection74: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t82, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t82] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t82, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:12,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:05,084 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 82 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:05,203 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection72: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t90, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t90] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t90, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:12,350 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,203 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 90 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:05,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,537 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 129 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:05,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:05,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 89 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:12,538 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 144 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,355 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_1 | 2019-10-15 03:15:05,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,355 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,085 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:12,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,355 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 90 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,355 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:05,370 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection73: begin an election at term 91 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:05,155 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:71ms, electionTimeout:71ms datanode_2 | 2019-10-15 03:15:05,374 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 84 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:12,538 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,538 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 82 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,634 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 130 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:05,374 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:12,634 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,390 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection73: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t91, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t91] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t91, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,390 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 91 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:12,635 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,172 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection75: begin an election at term 83 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:12,681 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 131 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,174 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 90 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:12,681 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,681 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,559 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_1 | 2019-10-15 03:15:05,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,681 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,174 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,559 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 91 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,699 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 145 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:05,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection75: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t83, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t83] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t83, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,704 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection74: begin an election at term 92 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,700 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 83 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:05,708 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 85 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:12,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:05,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:05,708 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,811 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 132 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:05,708 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:12,811 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,357 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | 2019-10-15 03:15:12,811 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:05,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 83 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,811 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,708 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:12,855 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 146 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:05,370 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection76: begin an election at term 84 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,732 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection74: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t92, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t92] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t92, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:12,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,374 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 91 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:12,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,733 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 92 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,855 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,733 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:05,374 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:12,982 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 147 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,733 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,374 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:12,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,773 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_1 | 2019-10-15 03:15:05,389 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection76: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t84, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t84] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t84, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:12,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:05,390 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 84 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:05,773 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 92 for changeToCandidate datanode_3 | 2019-10-15 03:15:12,982 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:05,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:13,008 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 133 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:05,796 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection75: begin an election at term 93 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,555 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_3 | 2019-10-15 03:15:13,009 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,817 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection75: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t93, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t93] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t93, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,555 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,009 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,817 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 93 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:05,556 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 84 for changeToCandidate datanode_3 | 2019-10-15 03:15:13,009 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:05,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:13,105 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 134 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:05,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,704 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection77: begin an election at term 85 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,869 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 86 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,707 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 92 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:13,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:05,869 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,707 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:13,106 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:05,981 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_3 | 2019-10-15 03:15:13,160 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 148 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:05,732 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection77: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t85, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t85] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t85, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:05,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,733 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 85 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:05,981 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 93 for changeToCandidate datanode_1 | 2019-10-15 03:15:05,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:13,160 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:05,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:05,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:13,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 149 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:05,987 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection76: begin an election at term 94 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,798 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 93 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:13,271 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,006 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection76: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t94, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t94] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t94, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:05,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 94 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:05,798 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:13,271 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:05,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:13,271 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,859 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_3 | 2019-10-15 03:15:13,323 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 135 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:06,102 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 87 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:05,859 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,323 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,860 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 85 for changeToCandidate datanode_2 | 2019-10-15 03:15:06,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:05,860 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:13,323 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,102 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,867 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection78: begin an election at term 86 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,323 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,111 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_1 | 2019-10-15 03:15:05,897 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection78: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t86, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t86] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t86, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,373 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 136 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:06,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,897 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 86 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 94 for changeToCandidate datanode_1 | 2019-10-15 03:15:05,897 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:13,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:05,897 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,124 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection77: begin an election at term 95 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,373 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,989 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 94 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,126 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 88 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:13,390 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 150 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:05,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,126 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,390 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:05,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,126 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:13,391 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,126 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:05,989 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:13,391 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,088 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_3 | 2019-10-15 03:15:13,417 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 137 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:06,143 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection77: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t95, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t95] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t95, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,417 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,144 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 95 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:06,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 86 for changeToCandidate datanode_3 | 2019-10-15 03:15:13,417 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:13,417 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:06,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:13,531 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 151 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:06,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection79: begin an election at term 87 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,251 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:15:13,531 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,251 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 95 for changeToCandidate datanode_3 | 2019-10-15 03:15:13,532 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,252 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 152 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:06,107 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection79: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t87, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,280 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection78: begin an election at term 96 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,124 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection79: begin an election at term 88 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 138 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:06,283 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 89 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,127 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 95 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,283 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,283 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,283 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,127 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:13,590 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,309 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection78: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t96, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t96] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t96, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,143 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection79: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t88, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t88] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t88, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,309 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 96 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:06,143 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 88 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,590 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,309 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:06,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:13,605 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 153 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:13,605 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,262 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_3 | 2019-10-15 03:15:13,605 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,450 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_1 | 2019-10-15 03:15:06,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,450 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 88 for changeToCandidate datanode_1 | 2019-10-15 03:15:06,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:06,450 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 96 for changeToCandidate datanode_2 | 2019-10-15 03:15:06,451 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:06,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection79: begin an election at term 97 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,495 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection79: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t97, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t97] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t97, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,605 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection80: begin an election at term 89 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,496 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 97 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,631 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 154 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:06,282 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 96 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:13,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,526 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 90 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:13,631 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,282 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,309 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection80: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t89, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t89] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t89, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,526 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:13,770 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 139 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:06,668 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_3 | 2019-10-15 03:15:13,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,310 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 89 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,668 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 97 for changeToCandidate datanode_1 | 2019-10-15 03:15:06,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:06,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:06,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,682 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection80: begin an election at term 98 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,770 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,480 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 97 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,709 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection80: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t98, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t98] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t98, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,883 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 155 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:06,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,883 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,709 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 98 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:13,883 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,480 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:13,883 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,507 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | 2019-10-15 03:15:06,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:13,909 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 140 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,760 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 91 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:13,909 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,507 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 89 for changeToCandidate datanode_2 | 2019-10-15 03:15:06,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:13,909 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:06,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:06,760 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,802 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | 2019-10-15 03:15:06,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,524 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection81: begin an election at term 90 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:13,909 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 98 for changeToCandidate datanode_1 | 2019-10-15 03:15:06,553 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection81: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t90, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t90] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t90, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 141 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,553 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 90 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:06,816 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection81: begin an election at term 99 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,553 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:06,844 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection81: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t99, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t99] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t99, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,553 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,844 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 99 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:14,115 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,685 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 98 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 156 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,845 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 92 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,116 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,685 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:14,176 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 157 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,845 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,743 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | 2019-10-15 03:15:14,176 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,905 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_1 | 2019-10-15 03:15:06,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,176 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,905 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,743 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 90 for changeToCandidate datanode_3 | 2019-10-15 03:15:14,176 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:06,905 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 99 for changeToCandidate datanode_3 | 2019-10-15 03:15:14,265 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 142 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,757 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection82: begin an election at term 91 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:06,905 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection82: begin an election at term 100 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,775 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection82: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t91, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t91] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t91, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,949 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection82: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t100, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t100] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t100, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,775 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 91 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:14,265 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:06,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 100 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:06,775 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:06,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:06,775 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:14,305 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 158 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:14,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,970 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_3 | 2019-10-15 03:15:14,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:06,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,306 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 99 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:06,970 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 100 for changeToCandidate datanode_3 | 2019-10-15 03:15:14,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 143 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:06,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:06,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:14,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:06,984 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection83: begin an election at term 101 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,818 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,013 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection83: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t101, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t101] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t101, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,367 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,013 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 101 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:06,824 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 144 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:06,824 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,824 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 91 for changeToCandidate datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 159 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:06,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:07,030 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 93 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:06,843 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection83: begin an election at term 92 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,864 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection83: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t92, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t92] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t92, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:14,536 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 92 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,083 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 94 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:14,536 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,083 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:14,564 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 145 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,083 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:06,922 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 100 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:14,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,083 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:06,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,201 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_1 | 2019-10-15 03:15:06,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:14,564 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:06,922 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:14,776 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 160 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:07,201 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 101 for changeToCandidate datanode_1 | 2019-10-15 03:15:06,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 101 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:14,776 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:06,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,776 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,215 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection84: begin an election at term 102 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:14,776 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,234 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection84: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t102, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t102] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t102, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:06,986 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:14,806 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 146 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 102 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:07,013 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_3 | 2019-10-15 03:15:14,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:14,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,807 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 92 for changeToCandidate datanode_3 | 2019-10-15 03:15:14,821 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 161 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:07,287 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 95 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,028 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection84: begin an election at term 93 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,288 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:14,821 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection84: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t93, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t93] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t93, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,288 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,057 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 93 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,349 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | 2019-10-15 03:15:07,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,057 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:14,850 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 147 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 102 for changeToCandidate datanode_1 | 2019-10-15 03:15:07,057 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:07,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:07,058 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_3 | 2019-10-15 03:15:14,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,362 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection85: begin an election at term 103 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,392 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection85: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t103, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t103] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t103, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 93 for changeToCandidate datanode_2 | 2019-10-15 03:15:07,392 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 103 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:14,850 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:14,946 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 162 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:07,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:07,080 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection85: begin an election at term 94 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:14,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,100 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection85: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t94, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t94] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t94, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:14,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,100 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 94 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,476 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 96 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:14,946 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,100 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:07,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,072 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 163 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,100 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:07,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 102 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:15,072 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,476 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,072 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,554 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_1 | 2019-10-15 03:15:07,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:07,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,072 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,219 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,074 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 148 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 103 for changeToCandidate datanode_1 | 2019-10-15 03:15:07,272 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:15,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:07,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,568 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection86: begin an election at term 104 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,272 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 94 for changeToCandidate datanode_3 | 2019-10-15 03:15:15,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,590 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection86: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t104, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t104] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t104, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:15,074 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,183 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 164 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,285 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection86: begin an election at term 95 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection86: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t95, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t95] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t95, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,590 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 104 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,314 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 95 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:15,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,607 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 97 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,314 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,184 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,314 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:15,210 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 149 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,365 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 103 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:15,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,607 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:15,210 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,760 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_1 | 2019-10-15 03:15:07,365 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,240 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 150 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,459 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_2 | 2019-10-15 03:15:07,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,760 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 104 for changeToCandidate datanode_3 | 2019-10-15 03:15:15,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 95 for changeToCandidate datanode_2 | 2019-10-15 03:15:07,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:15,241 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:15,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 165 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,473 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection87: begin an election at term 96 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,780 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection87: begin an election at term 105 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:15,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,782 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 98 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:15,271 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,504 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection87: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t96, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t96] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t96, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,271 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,504 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 96 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:15,304 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 166 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:07,782 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,809 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection87: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t105, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t105] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t105, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:15,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:07,810 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 105 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:15,304 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 167 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:07,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,847 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_1 | 2019-10-15 03:15:07,570 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 104 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,847 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,570 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,343 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,847 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 105 for changeToCandidate datanode_1 | 2019-10-15 03:15:07,570 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 151 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:07,571 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection88: begin an election at term 106 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,591 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_3 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,891 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection88: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t106, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t106] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t106, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,591 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,343 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,891 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 106 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:07,591 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 96 for changeToCandidate datanode_2 | 2019-10-15 03:15:07,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,481 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 152 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:07,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:15,481 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,605 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection88: begin an election at term 97 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,906 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_3 | 2019-10-15 03:15:15,481 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,622 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection88: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t97, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t97] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t97, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:07,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,481 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,622 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 97 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:07,906 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 106 for changeToCandidate datanode_3 | 2019-10-15 03:15:15,508 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 153 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:07,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:15,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:07,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection89: begin an election at term 107 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,754 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_3 | 2019-10-15 03:15:15,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,948 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection89: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t107, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t107] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t107, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,948 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 107 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:15,508 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,754 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 97 for changeToCandidate datanode_2 | 2019-10-15 03:15:07,948 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 168 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,780 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection89: begin an election at term 98 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:07,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 105 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 154 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:07,948 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,978 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 99 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,782 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:07,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:07,809 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection89: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t98, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t98] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t98, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:07,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,809 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 98 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:15,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,809 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:07,988 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:15:15,813 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 169 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,810 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:07,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 106 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:07,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 107 for changeToCandidate datanode_3 | 2019-10-15 03:15:15,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:07,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:15,813 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection90: begin an election at term 108 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:15,814 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 155 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:07,864 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,021 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection90: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t108, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t108] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t108, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:15,814 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 107 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,021 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 108 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:15,814 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:15,814 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:15,831 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 170 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,921 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 100 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,962 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:15:08,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:07,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:15,831 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:07,962 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 98 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:15,886 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 171 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:07,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:08,212 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_3 | 2019-10-15 03:15:15,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:07,976 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection90: begin an election at term 99 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:15,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,005 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection90: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t99, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t99] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t99, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,212 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 108 for changeToCandidate datanode_3 | 2019-10-15 03:15:15,886 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:08,005 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 99 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:16,166 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 156 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:08,226 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection91: begin an election at term 109 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,005 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:16,167 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,256 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection91: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t109, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t109] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t109, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,005 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:16,166 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 172 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,256 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 109 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:08,006 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 108 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:16,167 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:08,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,167 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:16,167 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,258 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_1 | 2019-10-15 03:15:08,006 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,167 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,087 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_3 | 2019-10-15 03:15:16,167 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,265 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 157 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 99 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,258 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 109 for changeToCandidate datanode_3 | 2019-10-15 03:15:16,265 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:08,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:16,265 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection91: begin an election at term 100 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,272 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection92: begin an election at term 110 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:16,265 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection91: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t100, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t100] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t100, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,298 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection92: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t110, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t110] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t110, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:16,303 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 158 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 100 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:08,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 110 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:16,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:08,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:16,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:16,303 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,299 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 101 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,229 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 109 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:16,393 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 173 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:16,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,230 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,393 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,299 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,271 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | 2019-10-15 03:15:16,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 159 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,271 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 100 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 102 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:16,440 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:08,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,274 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 110 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:08,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:16,440 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,594 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 174 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:08,274 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,462 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_3 | 2019-10-15 03:15:16,594 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,297 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection92: begin an election at term 101 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,594 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection92: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t101, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t101] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t101, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 110 for changeToCandidate datanode_3 | 2019-10-15 03:15:16,594 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,314 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 101 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:16,653 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 175 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:08,314 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:08,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:16,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,475 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection93: begin an election at term 111 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:16,654 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,412 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_2 | 2019-10-15 03:15:08,492 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection93: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t111, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t111] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t111, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:16,654 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,492 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 111 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:16,654 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 160 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,412 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 101 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,492 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:16,654 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:08,492 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:16,654 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,437 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection93: begin an election at term 102 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,553 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_3 | 2019-10-15 03:15:16,654 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,749 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 161 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,453 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection93: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t102, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t102] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t102, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,553 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 111 for changeToCandidate datanode_3 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:08,453 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 102 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:16,749 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 176 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,567 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection94: begin an election at term 112 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection94: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t112, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t112] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t112, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 112 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:08,477 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 111 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:08,477 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,749 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:16,750 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,478 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,750 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,632 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 103 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,570 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 112 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 177 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,570 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,570 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,570 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,632 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,612 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 162 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,612 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 102 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,793 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 112 for changeToCandidate datanode_3 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:08,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:08,630 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection94: begin an election at term 103 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,701 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection95: begin an election at term 113 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:16,794 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,647 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection94: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t103, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t103] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t103, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 104 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:16,873 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 163 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,647 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 103 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,647 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:16,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,703 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:16,873 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,687 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:15:16,915 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 178 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,731 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection95: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t113, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t113] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t113, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:16,915 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,731 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 113 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:08,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 103 for changeToCandidate datanode_3 | 2019-10-15 03:15:16,915 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,731 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:08,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:16,915 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,731 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,701 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection95: begin an election at term 104 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,000 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 179 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,787 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 105 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,703 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 113 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:17,000 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,787 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,000 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,787 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,703 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,000 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:08,787 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,703 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:17,016 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 164 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:08,797 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_1 | 2019-10-15 03:15:08,731 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection95: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t104, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t104] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t104, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,016 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,016 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 104 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:08,797 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 113 for changeToCandidate datanode_3 | 2019-10-15 03:15:17,016 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:08,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:17,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 180 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:08,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection96: begin an election at term 114 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,771 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:15:17,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,811 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection96: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t114, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t114] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t114, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,771 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 114 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:08,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:17,199 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 165 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,771 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 104 for changeToCandidate datanode_2 | 2019-10-15 03:15:08,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,771 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:17,199 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,956 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 106 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,785 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection96: begin an election at term 105 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,199 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:08,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,199 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection96: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t105, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t105] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t105, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:08,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:17,247 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 181 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,957 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:17,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 105 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:08,975 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_3 | 2019-10-15 03:15:17,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:08,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,247 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,249 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 166 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,805 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 114 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:08,975 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 114 for changeToCandidate datanode_3 | 2019-10-15 03:15:17,249 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:08,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:17,249 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:08,982 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection97: begin an election at term 115 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,249 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:08,805 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection97: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t115, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t115] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t115, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,281 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 167 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,939 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | 2019-10-15 03:15:09,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 115 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:17,281 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:17,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:08,939 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 105 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:17,282 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,125 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_1 | 2019-10-15 03:15:08,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:17,399 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 168 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:08,954 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection97: begin an election at term 106 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,399 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:08,981 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection97: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t106, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t106] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t106, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,399 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 115 for changeToCandidate datanode_1 | 2019-10-15 03:15:08,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 106 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:17,399 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:08,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:17,448 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 182 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection98: begin an election at term 116 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,448 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,163 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection98: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t116, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t116] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t116, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:08,985 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 115 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:17,448 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,163 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 116 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:08,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,448 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,163 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:08,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,758 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 169 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:09,163 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:08,985 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:17,758 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 183 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,143 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 116 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,184 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 107 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,184 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:09,144 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:17,758 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,218 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_1 | 2019-10-15 03:15:09,170 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,170 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,218 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 116 for changeToCandidate datanode_3 | 2019-10-15 03:15:17,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,170 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 106 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:17,914 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 184 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,231 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection99: begin an election at term 117 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,170 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:17,914 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,250 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection99: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t117, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t117] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t117, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection98: begin an election at term 107 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,914 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 117 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,200 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection98: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t107, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t107] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t107, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,914 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:09,200 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 107 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:17,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 170 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:09,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:09,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:17,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,273 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_1 | 2019-10-15 03:15:09,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:17,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,234 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 117 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:17,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,273 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 117 for changeToCandidate datanode_1 | 2019-10-15 03:15:09,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:17,976 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 171 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:09,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:09,234 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:17,976 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,287 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection100: begin an election at term 118 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,976 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,303 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection100: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t118, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t118] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t118, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:17,976 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 118 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 118 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:18,118 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 172 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:09,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,378 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 108 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:18,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 185 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,361 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_2 | 2019-10-15 03:15:09,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,378 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 107 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_1 | 2019-10-15 03:15:09,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,376 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection99: begin an election at term 108 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 118 for changeToCandidate datanode_1 | 2019-10-15 03:15:09,404 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection99: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t108, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t108] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t108, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:18,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 109 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,404 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 108 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 186 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 173 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:09,499 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,512 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection101: begin an election at term 119 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,457 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | 2019-10-15 03:15:09,528 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection101: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t119, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t119] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t119, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,457 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,160 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,457 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 108 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,528 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 119 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,457 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:09,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection100: begin an election at term 109 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,160 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,513 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection100: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t109, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t109] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t109, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,567 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | 2019-10-15 03:15:18,282 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 174 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 109 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:09,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:18,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,567 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 119 for changeToCandidate datanode_1 | 2019-10-15 03:15:09,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:18,282 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:09,514 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 119 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:18,296 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 175 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:09,582 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection102: begin an election at term 120 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,296 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection102: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t120, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t120] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t120, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:18,296 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 120 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,514 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,296 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:09,584 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 120 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:18,312 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 176 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,312 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:18,312 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:09,585 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,312 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,644 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 110 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,637 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_3 | 2019-10-15 03:15:18,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 187 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,644 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,644 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:09,637 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 109 for changeToCandidate datanode_3 | 2019-10-15 03:15:18,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,644 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,366 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:09,686 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | 2019-10-15 03:15:18,390 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 177 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,641 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection101: begin an election at term 110 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,686 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,391 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection101: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t110, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t110] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t110, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,686 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 120 for changeToCandidate datanode_3 | 2019-10-15 03:15:18,391 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 110 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:09,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:18,391 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:09,711 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection103: begin an election at term 121 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:18,462 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 188 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 111 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,684 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_3 | 2019-10-15 03:15:18,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,462 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,714 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,596 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 178 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,685 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 110 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,740 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection103: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t121, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t121] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t121, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:18,596 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,740 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 121 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:18,596 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:09,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:18,596 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection102: begin an election at term 111 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,648 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 189 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 121 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,824 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 112 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:18,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,824 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,713 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,824 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,648 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,824 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,751 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 190 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,740 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection102: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t111, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t111] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t111, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,841 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:15:18,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,740 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 111 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:09,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:09,841 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 121 for changeToCandidate datanode_3 | 2019-10-15 03:15:18,751 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:18,803 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 179 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,807 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:09,856 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection104: begin an election at term 122 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:18,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,807 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 111 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,873 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection104: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t122, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:18,803 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,821 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection103: begin an election at term 112 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection104: begin an election at term 123 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,839 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection103: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t112, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t112] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t112, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,916 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 113 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:18,839 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 191 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,839 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 112 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,839 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:09,916 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,839 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,916 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:18,839 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,858 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 122 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,936 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection104: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t123, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t123] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t123, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 180 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,858 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,936 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 123 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,858 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:09,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,858 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:09,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:18,970 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,879 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_2 | 2019-10-15 03:15:09,954 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 192 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:09,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:09,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,879 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 112 for changeToCandidate datanode_2 | 2019-10-15 03:15:09,954 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 123 for changeToCandidate datanode_3 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:09,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:09,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:18,971 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,914 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection104: begin an election at term 113 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:09,984 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection105: begin an election at term 124 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 193 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:09,996 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection105: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t124, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,009 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection105: begin an election at term 125 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,919 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 123 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:19,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,026 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection105: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t125, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t125] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t125, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,919 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,026 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 125 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:19,013 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 194 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:10,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:09,919 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,105 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 181 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,919 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,105 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,936 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection104: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t113, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t113] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t113, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,105 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 125 for changeToCandidate datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,936 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 113 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,105 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,119 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection106: begin an election at term 126 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,936 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,059 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,152 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection106: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t126, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t126] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t126, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:09,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:19,059 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,152 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 126 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:09,987 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 124 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,149 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 182 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:09,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:09,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,153 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 114 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:09,987 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,151 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,011 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 125 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:10,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,213 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 195 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,213 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:19,213 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,259 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:15:10,011 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:19,213 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,123 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 126 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:19,247 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 183 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:10,259 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 126 for changeToCandidate datanode_1 | 2019-10-15 03:15:10,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:10,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection107: begin an election at term 127 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,123 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,318 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection107: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t127, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t127] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t127, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,248 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,129 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | 2019-10-15 03:15:10,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 127 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:19,294 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 196 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,129 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 113 for changeToCandidate datanode_2 | 2019-10-15 03:15:10,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:10,321 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 115 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,294 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,412 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 184 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:10,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,412 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,150 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection105: begin an election at term 114 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,321 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:19,412 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,168 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection105: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t114, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t114] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t114, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,469 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 116 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,412 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,496 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 197 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,168 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 114 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:10,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,496 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:19,496 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | 2019-10-15 03:15:19,496 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,305 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 127 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:19,579 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 185 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:10,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 127 for changeToCandidate datanode_1 | 2019-10-15 03:15:10,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:19,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,522 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection108: begin an election at term 128 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,305 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection108: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t128, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t128] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t128, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,579 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,309 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | 2019-10-15 03:15:10,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 128 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:19,617 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 198 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,309 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 114 for changeToCandidate datanode_2 | 2019-10-15 03:15:10,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:10,586 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 117 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,617 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,316 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection106: begin an election at term 115 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,639 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 199 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,327 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection106: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t115, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t115] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t115, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,327 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 115 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,586 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:10,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | 2019-10-15 03:15:19,639 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,640 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 186 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:10,451 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | 2019-10-15 03:15:10,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 128 for changeToCandidate datanode_3 | 2019-10-15 03:15:19,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:10,451 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 115 for changeToCandidate datanode_3 | 2019-10-15 03:15:19,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,696 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection109: begin an election at term 129 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,640 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,728 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection109: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t129, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t129] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t129, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,759 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 200 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:10,728 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 129 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:10,466 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection107: begin an election at term 116 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:10,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection107: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t116, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t116] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t116, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,764 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 118 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,801 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 201 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,496 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 116 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:10,764 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:19,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,829 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 119 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:19,802 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,525 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 128 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:10,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,852 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 187 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:10,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:19,852 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:19,852 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,526 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,829 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:19,852 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,571 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | 2019-10-15 03:15:19,890 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 188 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:10,851 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:121ms datanode_1 | 2019-10-15 03:15:10,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 116 for changeToCandidate datanode_2 | 2019-10-15 03:15:10,851 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 129 for changeToCandidate datanode_3 | 2019-10-15 03:15:19,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:10,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:10,584 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection108: begin an election at term 117 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,855 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection110: begin an election at term 130 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:10,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection108: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t117, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t117] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t117, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,878 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection110: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t130, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t130] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t130, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:19,890 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 117 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,878 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 130 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:19,987 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 189 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:10,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:10,878 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:19,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:10,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 129 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:19,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:10,949 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_1 | 2019-10-15 03:15:10,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:19,987 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 130 for changeToCandidate datanode_3 | 2019-10-15 03:15:20,024 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 202 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,699 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:20,024 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,754 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:15:10,955 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection111: begin an election at term 131 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,024 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,964 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection111: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t131, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t131] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t131, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,024 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,754 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 117 for changeToCandidate datanode_2 | 2019-10-15 03:15:10,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 131 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:10,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:20,229 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 190 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:10,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:10,762 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection109: begin an election at term 118 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:20,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,770 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection109: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t118, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t118] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t118, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:10,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 120 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:20,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,771 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 118 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:10,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,771 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:20,229 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:10,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:10,771 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:10,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:20,238 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 203 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,821 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | 2019-10-15 03:15:10,991 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_3 | 2019-10-15 03:15:20,238 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:10,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:20,238 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,821 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 118 for changeToCandidate datanode_2 | 2019-10-15 03:15:10,991 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 131 for changeToCandidate datanode_1 | 2019-10-15 03:15:10,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:10,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:20,238 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,827 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection110: begin an election at term 119 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,387 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 191 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,020 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection112: begin an election at term 132 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:10,843 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection110: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t119, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t119] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t119, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,844 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 119 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:11,039 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection112: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t132, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t132] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t132, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,039 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 132 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:20,387 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:20,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 204 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:11,085 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_1 | 2019-10-15 03:15:10,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:11,085 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:20,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,085 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 132 for changeToCandidate datanode_3 | 2019-10-15 03:15:20,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,085 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:20,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,857 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 130 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,110 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection113: begin an election at term 133 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,589 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 192 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:10,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,137 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection113: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t133, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t133] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t133, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,589 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,137 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 133 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:20,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,857 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:20,590 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 131 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:10,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,152 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_1 | 2019-10-15 03:15:10,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:20,697 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 205 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:10,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,152 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 133 for changeToCandidate datanode_3 | 2019-10-15 03:15:20,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:20,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:10,967 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | 2019-10-15 03:15:11,178 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection114: begin an election at term 134 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:10,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,180 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 121 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:10,967 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 119 for changeToCandidate datanode_2 | 2019-10-15 03:15:11,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:10,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:20,806 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 193 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:10,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection111: begin an election at term 120 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,180 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection111: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t120, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t120] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t120, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection114: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t134, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t134] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t134, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,021 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 120 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:20,806 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,196 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 134 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:20,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 206 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:11,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:20,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,022 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 132 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,384 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:20,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,022 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,384 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:20,815 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,022 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,384 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 134 for changeToCandidate datanode_1 | 2019-10-15 03:15:11,022 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:20,959 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 207 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,384 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:11,112 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 133 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,396 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection115: begin an election at term 135 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:20,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,396 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 122 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:20,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:20,959 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,112 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:21,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 194 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,163 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_2 | 2019-10-15 03:15:11,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,424 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection115: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t135, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t135] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t135, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,163 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 120 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,424 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 135 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:21,007 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,178 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection112: begin an election at term 121 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,061 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 208 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:11,180 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 134 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:21,061 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,061 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:21,061 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,180 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,426 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 123 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 209 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:11,195 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection112: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t121, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t121] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t121, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:11,426 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,195 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 121 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:11,426 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:11,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:11,426 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 195 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,443 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_1 | 2019-10-15 03:15:11,369 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,369 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,108 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,443 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 135 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,109 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:11,369 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 121 for changeToCandidate datanode_2 | 2019-10-15 03:15:11,457 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection116: begin an election at term 136 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:21,226 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 196 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:21,227 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,369 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:11,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection116: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t136, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,227 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,394 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection113: begin an election at term 122 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:11,488 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection116: begin an election at term 137 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,227 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 135 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 197 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,506 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection116: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t137, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t137] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t137, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 210 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,506 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 137 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,399 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:11,399 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:11,401 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection113: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t122, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,646 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 124 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,423 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection113: begin an election at term 123 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:11,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:21,300 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:21,300 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,440 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection113: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t123, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t123] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t123, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,316 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 198 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,669 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_1 | 2019-10-15 03:15:11,440 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 123 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:21,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:21,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,669 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 137 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,316 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,459 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 136 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:21,353 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 211 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:11,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,724 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection117: begin an election at term 138 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,752 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection117: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t138, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t138] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t138, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,459 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:21,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,752 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 138 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,489 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 137 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:21,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:11,489 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,353 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:11,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:21,368 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 199 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,814 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_1 | 2019-10-15 03:15:11,490 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:21,368 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,629 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:21,369 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 138 for changeToCandidate datanode_1 | 2019-10-15 03:15:11,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,369 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:11,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 123 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,469 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 200 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,829 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection118: begin an election at term 139 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:21,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,643 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection114: begin an election at term 124 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,713 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection114: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t124, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t124] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t124, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:11,848 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection118: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t139, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,469 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 124 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:21,547 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 201 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:11,873 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection118: begin an election at term 140 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:21,547 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,910 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection118: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t140, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t140] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t140, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:21,547 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,910 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 140 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,726 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 138 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:21,547 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:11,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,548 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 212 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:11,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:11,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 125 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:11,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,726 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:11,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:11,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 139 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:11,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,548 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,008 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 213 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:11,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,009 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,832 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,009 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 140 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,009 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:11,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 140 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:21,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,035 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 126 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 202 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:21,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:11,875 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,036 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:11,903 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_2 | 2019-10-15 03:15:12,049 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection119: begin an election at term 141 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,830 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 214 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:11,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,067 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection119: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t141, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 124 for changeToCandidate datanode_3 | 2019-10-15 03:15:21,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection119: begin an election at term 142 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:21,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,097 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection119: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t142, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t142] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t142, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:11,950 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection115: begin an election at term 125 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,830 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,097 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 142 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:11,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection115: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t125, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t125] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t125, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:21,885 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 203 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,097 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:11,978 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 125 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:21,885 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,097 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:11,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:21,885 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,155 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_1 | 2019-10-15 03:15:11,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:21,885 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:11,995 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_3 | 2019-10-15 03:15:22,003 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 215 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:12,155 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 142 for changeToCandidate datanode_1 | 2019-10-15 03:15:11,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,003 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:11,995 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 125 for changeToCandidate datanode_3 | 2019-10-15 03:15:22,003 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,249 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 127 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:11,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:22,003 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,033 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection116: begin an election at term 126 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,044 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 204 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,249 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,051 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 141 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:22,044 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,044 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,271 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection120: begin an election at term 143 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:22,044 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,051 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,219 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 216 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:12,325 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection120: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t143, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t143] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t143, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,074 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection116: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t126, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t126] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t126, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 143 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 126 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:22,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:12,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:22,220 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,326 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:22,234 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 205 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 128 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,091 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 142 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:22,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:22,234 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,091 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,291 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 217 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:12,519 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_1 | 2019-10-15 03:15:12,105 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | 2019-10-15 03:15:22,291 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,519 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,519 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 143 for changeToCandidate datanode_1 | 2019-10-15 03:15:12,105 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 126 for changeToCandidate datanode_3 | 2019-10-15 03:15:22,292 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,520 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:12,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:22,338 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 206 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,535 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection121: begin an election at term 144 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection117: begin an election at term 127 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,338 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,537 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 129 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,272 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection117: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t127, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t127] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t127, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,338 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 127 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:22,338 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:22,449 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 207 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,537 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 143 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:12,554 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection121: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t144, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t144] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t144, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 144 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:22,450 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:22,452 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 218 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:12,273 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,339 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:12,634 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 130 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:22,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,340 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,340 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 127 for changeToCandidate datanode_2 | 2019-10-15 03:15:12,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:22,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,635 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,453 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,340 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:12,681 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 131 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:22,578 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 208 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,347 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection118: begin an election at term 128 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,397 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection118: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t128, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t128] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t128, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:22,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 128 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:12,681 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,578 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:12,693 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | 2019-10-15 03:15:22,613 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 219 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:12,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,613 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,523 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | 2019-10-15 03:15:12,693 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 144 for changeToCandidate datanode_3 | 2019-10-15 03:15:22,613 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:12,523 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 128 for changeToCandidate datanode_2 | 2019-10-15 03:15:12,697 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection122: begin an election at term 145 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,613 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:22,807 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 209 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,714 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection122: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t145, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t145] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t145, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection119: begin an election at term 129 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,807 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 145 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,537 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 144 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:22,807 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:22,808 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:22,823 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 220 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:12,537 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 132 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,553 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection119: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t129, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t129] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t129, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:22,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,554 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 129 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:22,823 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:22,842 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 221 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:12,554 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:12,811 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:22,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,554 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:12,838 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_3 | 2019-10-15 03:15:22,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,606 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | 2019-10-15 03:15:12,838 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:22,843 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,839 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 145 for changeToCandidate datanode_1 | 2019-10-15 03:15:12,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:12,606 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 129 for changeToCandidate datanode_3 | 2019-10-15 03:15:22,912 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 222 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:12,851 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection123: begin an election at term 146 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,607 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:22,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:12,870 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection123: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t146, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t146] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t146, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,632 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection120: begin an election at term 130 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,871 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 146 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,670 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection120: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t130, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:22,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:12,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,679 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection120: begin an election at term 131 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,958 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:15:22,912 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:12,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 210 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:12,958 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 146 for changeToCandidate datanode_3 | 2019-10-15 03:15:23,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection120: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t131, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t131] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t131, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:12,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:23,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:12,695 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 131 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:12,979 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection124: begin an election at term 147 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,006 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:13,007 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection124: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t147, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t147] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t147, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,047 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 223 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:12,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 145 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:23,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,007 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 147 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,047 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:12,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,008 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:23,113 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 211 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,699 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 133 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,113 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,795 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | 2019-10-15 03:15:13,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,795 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,113 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,795 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 131 for changeToCandidate datanode_3 | 2019-10-15 03:15:23,114 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,010 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,795 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:23,154 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 224 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 134 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,154 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,809 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection121: begin an election at term 132 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,154 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:12,833 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection121: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t132, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t132] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t132, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,154 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,253 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 212 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:13,137 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:15:13,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 132 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:23,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,137 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 147 for changeToCandidate datanode_1 | 2019-10-15 03:15:12,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:23,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:12,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,253 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,157 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection125: begin an election at term 148 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,855 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 146 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:23,306 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 225 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,174 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection125: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t148, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t148] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t148, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,174 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 148 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:12,855 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:23,306 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:12,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 147 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,254 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_3 | 2019-10-15 03:15:23,444 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 213 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:12,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,254 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,444 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:12,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,444 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,255 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 148 for changeToCandidate datanode_1 | 2019-10-15 03:15:12,982 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,444 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,255 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:12,983 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_3 | 2019-10-15 03:15:23,446 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 226 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,268 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection126: begin an election at term 149 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,446 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,285 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection126: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t149, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t149] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t149, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:12,983 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 132 for changeToCandidate datanode_3 | 2019-10-15 03:15:23,446 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,285 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 149 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:12,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:23,446 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,285 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:13,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection122: begin an election at term 133 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,517 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 227 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,285 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection122: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t133, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t133] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t133, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,517 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,323 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 135 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 133 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:23,517 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:23,517 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,087 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | 2019-10-15 03:15:13,323 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,538 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 214 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,372 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 136 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 133 for changeToCandidate datanode_1 | 2019-10-15 03:15:13,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:13,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,101 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection123: begin an election at term 134 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,373 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,129 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection123: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t134, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t134] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t134, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,373 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_3 | 2019-10-15 03:15:23,538 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,129 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 134 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:23,704 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 228 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,373 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 149 for changeToCandidate datanode_3 | 2019-10-15 03:15:23,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:13,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:23,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,388 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection127: begin an election at term 150 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,704 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 215 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,414 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection127: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t150, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t150] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t150, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,159 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 148 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:23,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,414 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 150 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:13,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:13,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:23,848 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 216 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:13,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:23,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,417 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 137 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:23,848 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:13,417 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 217 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 229 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,515 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,515 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,159 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,515 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 150 for changeToCandidate datanode_1 | 2019-10-15 03:15:13,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 149 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,515 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,529 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection128: begin an election at term 151 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,881 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,546 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection128: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t151, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t151] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t151, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:23,881 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:23,999 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 218 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,306 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_2 | 2019-10-15 03:15:13,546 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 151 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:23,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,546 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:23,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,546 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:23,999 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,015 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 219 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:13,570 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_1 | 2019-10-15 03:15:13,306 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 134 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,015 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:13,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection124: begin an election at term 135 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,570 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,015 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,338 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection124: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t135, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t135] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t135, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,570 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 151 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,015 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 135 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:13,571 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:24,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 230 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:13,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:13,587 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: begin an election at term 152 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,589 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 138 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:24,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,357 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_2 | 2019-10-15 03:15:13,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:24,120 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 231 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:13,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 135 for changeToCandidate datanode_2 | 2019-10-15 03:15:13,590 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:13,595 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t152, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,370 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection125: begin an election at term 136 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,603 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: begin an election at term 153 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,120 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,389 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection125: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t136, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t136] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t136, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,603 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t153, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,389 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 136 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:13,629 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: begin an election at term 154 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:24,157 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 220 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:13,651 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection129: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t154, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t154] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t154, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,651 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 154 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:13,390 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 150 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:24,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:13,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,157 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 139 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,390 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,346 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 232 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,411 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_3 | 2019-10-15 03:15:24,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,770 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,411 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 136 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,346 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,850 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_1 | 2019-10-15 03:15:13,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:13,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,348 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 221 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,415 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection126: begin an election at term 137 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,851 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 154 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:24,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:13,880 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection130: begin an election at term 155 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,348 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,432 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection126: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t137, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t137] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t137, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:13,908 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection130: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t155, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t155] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t155, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,509 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 233 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:13,432 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 137 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:13,908 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 155 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:24,509 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:13,908 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:24,510 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,908 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:24,510 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,531 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 151 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:13,909 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 140 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:24,542 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 222 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:13,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,542 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:13,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:24,542 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,531 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:13,910 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,543 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,561 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:15:14,040 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_1 | 2019-10-15 03:15:13,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 137 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,582 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 234 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,040 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 155 for changeToCandidate datanode_1 | 2019-10-15 03:15:13,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:14,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:24,582 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,587 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection127: begin an election at term 138 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,582 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,112 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection131: begin an election at term 156 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,589 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 152 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:24,582 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,115 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 141 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,683 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 223 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,683 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,590 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,683 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,115 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,604 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection127: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t138, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t138] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t138, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,683 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,734 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 235 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,131 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection131: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t156, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t156] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t156, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 138 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:24,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,131 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 156 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:24,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,131 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 153 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:24,734 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,131 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,769 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 224 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,148 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,605 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,769 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 154 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,148 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 156 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,876 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 236 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:13,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:24,876 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,876 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,173 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection132: begin an election at term 157 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,631 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,876 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,264 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection132: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t157, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t157] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t157, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,741 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | 2019-10-15 03:15:24,889 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 225 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,264 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 157 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:13,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,889 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:13,741 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 138 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,889 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:24,889 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,265 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 142 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,768 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection128: begin an election at term 139 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,912 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 237 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:24,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,265 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,912 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,271 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_1 | 2019-10-15 03:15:13,797 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection128: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t139, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t139] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t139, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:14,271 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,943 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 238 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:13,797 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 139 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:14,271 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 157 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,943 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:14,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:24,944 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:14,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection133: begin an election at term 158 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:24,944 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,874 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_3 | 2019-10-15 03:15:24,978 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 239 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,320 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection133: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t158, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t158] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t158, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:13,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,978 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,320 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 158 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:13,874 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 139 for changeToCandidate datanode_3 | 2019-10-15 03:15:24,978 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:14,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:24,978 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,882 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 155 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:24,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 226 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,367 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 143 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,882 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:24,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,367 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:24,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,367 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:13,883 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:24,980 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,907 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection129: begin an election at term 140 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,152 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 227 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:13,929 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection129: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t140, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t140] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t140, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:14,367 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:25,152 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:13,929 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 140 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:25,152 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:13,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:14,501 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_3 | 2019-10-15 03:15:25,152 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:13,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:14,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,184 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 240 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:14,081 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,501 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 158 for changeToCandidate datanode_3 | 2019-10-15 03:15:25,184 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:25,282 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 228 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,533 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection134: begin an election at term 159 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,081 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 140 for changeToCandidate datanode_3 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,536 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 144 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,112 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection130: begin an election at term 141 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,115 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 156 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:25,282 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,536 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,297 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 241 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,563 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection134: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t159, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t159] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t159, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:25,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 159 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:14,115 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:25,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:14,131 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection130: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t141, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t141] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t141, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,297 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,131 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 141 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:25,458 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 229 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:25,458 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,564 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 145 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:25,458 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 157 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:25,458 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,485 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 242 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,564 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,742 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_1 | 2019-10-15 03:15:14,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,742 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,176 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:25,485 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,743 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 159 for changeToCandidate datanode_3 | 2019-10-15 03:15:25,556 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 230 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,743 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:14,770 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection135: begin an election at term 160 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,184 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_3 | 2019-10-15 03:15:25,557 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 243 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection135: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t160, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,807 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 146 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,184 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 141 for changeToCandidate datanode_3 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,263 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection131: begin an election at term 142 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,557 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,807 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,304 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection131: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t142, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t142] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t142, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,819 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection135: begin an election at term 161 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,304 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 158 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,847 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection135: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t161, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t161] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t161, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,557 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,847 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 161 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:14,304 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 142 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:25,619 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 231 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:14,847 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:14,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:25,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:25,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,850 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 147 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:25,620 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,305 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:25,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 244 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:14,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,349 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_3 | 2019-10-15 03:15:25,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,851 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,350 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 142 for changeToCandidate datanode_2 | 2019-10-15 03:15:14,930 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_3 | 2019-10-15 03:15:25,677 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,350 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:14,930 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,855 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 232 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,364 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection132: begin an election at term 143 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,930 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 161 for changeToCandidate datanode_1 | 2019-10-15 03:15:14,380 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection132: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t143, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t143] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t143, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:14,930 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:14,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 143 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:25,855 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:14,944 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection136: begin an election at term 162 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,905 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 245 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:14,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:14,953 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection136: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t162, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t162] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t162, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:25,905 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:14,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 162 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:14,510 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:15:14,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:25,905 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:14,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:25,905 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,510 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 143 for changeToCandidate datanode_2 | 2019-10-15 03:15:15,043 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_3 | 2019-10-15 03:15:25,942 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 233 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,043 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:25,942 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,533 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection133: begin an election at term 144 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,043 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 162 for changeToCandidate datanode_1 | 2019-10-15 03:15:14,535 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 159 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,044 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:15,069 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection137: begin an election at term 163 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,074 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 148 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:25,942 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:25,942 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,074 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,093 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 234 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,074 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:26,093 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,075 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,536 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,093 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,087 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection137: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t163, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t163] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t163, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,537 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection133: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t144, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,093 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,087 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 163 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:14,562 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection133: begin an election at term 145 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,132 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 246 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:14,590 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection133: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t145, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t145] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t145, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,590 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 145 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:26,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,167 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_1 | 2019-10-15 03:15:14,590 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,132 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,167 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,590 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,168 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 163 for changeToCandidate datanode_3 | 2019-10-15 03:15:26,248 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 235 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,168 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:26,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,181 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection138: begin an election at term 164 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,209 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection138: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t164, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t164] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t164, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,209 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 164 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:26,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,776 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 160 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,248 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:26,264 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 247 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,210 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 149 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,776 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,264 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,264 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,790 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_3 | 2019-10-15 03:15:26,264 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,278 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 248 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:14,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 145 for changeToCandidate datanode_2 | 2019-10-15 03:15:15,211 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,278 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 150 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:26,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection134: begin an election at term 146 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,279 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:14,820 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection134: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t146, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t146] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t146, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 236 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,820 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 146 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:26,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,240 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 249 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,371 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,253 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:43ms, electionTimeout:43ms datanode_3 | 2019-10-15 03:15:26,371 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,821 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 161 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,253 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 164 for changeToCandidate datanode_3 | 2019-10-15 03:15:26,371 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,371 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,268 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection139: begin an election at term 165 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:26,386 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 250 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,277 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection139: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t165, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:14,821 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection139: begin an election at term 166 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,835 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_2 | 2019-10-15 03:15:15,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection139: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t166, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t166] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t166, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,387 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,312 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 166 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:26,387 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 237 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 146 for changeToCandidate datanode_2 | 2019-10-15 03:15:15,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,315 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_1 | 2019-10-15 03:15:14,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:26,387 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,848 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection135: begin an election at term 147 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,579 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 238 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 166 for changeToCandidate datanode_1 | 2019-10-15 03:15:14,877 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection135: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t147, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t147] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t147, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:26,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,877 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 147 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:15,341 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection140: begin an election at term 167 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,579 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:15,343 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 151 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:26,593 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 251 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,343 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,593 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:14,945 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 162 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:26,593 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:14,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,593 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:14,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,370 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection140: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t167, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t167] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t167, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:26,716 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 239 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:14,946 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,370 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 167 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:26,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,064 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_2 | 2019-10-15 03:15:15,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:15,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,716 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,064 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 147 for changeToCandidate datanode_3 | 2019-10-15 03:15:26,832 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 252 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,481 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 152 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:26,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,071 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection136: begin an election at term 148 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,481 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,833 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,072 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 163 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,481 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:26,833 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,481 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,852 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 253 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,508 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 153 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:26,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,072 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,087 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection136: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t148, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t148] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t148, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:26,853 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,087 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 148 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:15,508 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,538 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_1 | 2019-10-15 03:15:15,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:26,854 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 240 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,538 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:26,854 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,183 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 164 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:26,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,538 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 167 for changeToCandidate datanode_1 | 2019-10-15 03:15:15,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,855 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,538 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:15,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:26,970 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 241 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,584 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection141: begin an election at term 168 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:15,184 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:26,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,184 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_2 | 2019-10-15 03:15:15,586 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 154 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:26,970 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:26,970 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,184 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 148 for changeToCandidate datanode_2 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,090 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 254 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,586 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:27,090 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,208 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection137: begin an election at term 149 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,612 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection141: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t168, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t168] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t168, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,090 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,612 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 168 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:27,090 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,232 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection137: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t149, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 255 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,238 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection137: begin an election at term 150 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,612 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:27,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,267 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection137: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t150, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t150] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t150, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,612 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,267 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 150 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:15,784 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:27,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:27,171 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 256 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:27,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,784 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 168 for changeToCandidate datanode_1 | 2019-10-15 03:15:15,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 165 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:27,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:27,171 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection142: begin an election at term 169 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,193 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 242 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 155 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:27,193 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,304 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 166 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:27,193 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,193 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,281 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 243 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,304 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,814 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:27,281 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,825 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection142: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t169, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:15,323 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_3 | 2019-10-15 03:15:27,281 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,830 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection142: begin an election at term 170 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,281 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,323 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 150 for changeToCandidate datanode_3 | 2019-10-15 03:15:27,359 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 257 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,858 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection142: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t170, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t170] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t170, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,858 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 170 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:15,341 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection138: begin an election at term 151 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 167 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:27,360 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,413 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 244 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,871 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_3 | 2019-10-15 03:15:27,413 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,343 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:15,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,413 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,370 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection138: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t151, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t151] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t151, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:15,871 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 170 for changeToCandidate datanode_3 | 2019-10-15 03:15:27,413 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 151 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:15,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:27,478 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 258 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:15,884 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection143: begin an election at term 171 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:15,902 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection143: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t171, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t171] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t171, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,464 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_2 | 2019-10-15 03:15:15,902 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 171 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:27,478 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:15,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 151 for changeToCandidate datanode_3 | 2019-10-15 03:15:27,557 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 259 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:15,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:15,959 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_3 | 2019-10-15 03:15:27,557 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,478 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection139: begin an election at term 152 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,557 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection139: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t152, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,596 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 245 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:15,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,596 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:15,960 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 171 for changeToCandidate datanode_1 | 2019-10-15 03:15:15,506 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection139: begin an election at term 153 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,596 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:15,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:16,163 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection144: begin an election at term 172 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,596 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,522 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection139: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t153, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t153] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t153, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,166 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 156 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:27,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 246 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,522 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 153 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,612 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:15,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:27,612 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,166 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,561 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_2 | 2019-10-15 03:15:16,204 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection144: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t172, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t172] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t172, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,612 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,204 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 172 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:15,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 153 for changeToCandidate datanode_3 | 2019-10-15 03:15:27,753 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 260 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:27,753 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:16,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:27,753 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,584 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection140: begin an election at term 154 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,265 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 157 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:27,753 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,586 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 168 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:27,780 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 247 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:16,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,780 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,265 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:27,780 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 158 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,586 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:27,780 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,857 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 261 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:15,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection140: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t154, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t154] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t154, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,857 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,303 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,612 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 154 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:27,857 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,376 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:27,857 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 248 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,806 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_2 | 2019-10-15 03:15:16,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 172 for changeToCandidate datanode_3 | 2019-10-15 03:15:27,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:27,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 154 for changeToCandidate datanode_2 | 2019-10-15 03:15:16,390 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection145: begin an election at term 173 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:16,408 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection145: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t173, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t173] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t173, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:27,995 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 262 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,812 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection141: begin an election at term 155 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,408 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 173 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:27,995 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,812 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 169 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,408 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:27,995 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:27,995 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,813 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,408 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,052 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 249 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,813 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 159 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:15,831 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection141: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t155, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t155] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t155, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,052 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,052 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 155 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,052 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,440 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,241 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 263 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,578 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_3 | 2019-10-15 03:15:28,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 170 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,578 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 173 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,241 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:15,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:28,245 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 250 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:16,592 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection146: begin an election at term 174 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:15,832 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,245 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,886 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 171 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,609 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection146: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t174, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t174] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t174, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,245 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:15,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,245 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,609 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 174 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:15,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:28,299 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 264 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,609 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:28,299 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:15,886 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,299 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,609 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:16,002 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:28,299 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,625 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_1 | 2019-10-15 03:15:16,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,314 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 265 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,625 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,003 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 155 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,625 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 174 for changeToCandidate datanode_1 | 2019-10-15 03:15:16,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:28,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,625 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:28,315 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,163 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection142: begin an election at term 156 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,651 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection147: begin an election at term 175 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,423 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 251 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,166 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 172 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,654 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 160 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:28,423 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,424 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,166 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,654 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,497 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 266 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,204 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection142: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t156, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t156] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t156, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,712 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection147: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t175, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,497 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 156 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,747 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection147: begin an election at term 176 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,497 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,749 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 161 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:28,497 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,238 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_3 | 2019-10-15 03:15:28,634 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 252 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,238 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,634 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,749 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,238 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 156 for changeToCandidate datanode_2 | 2019-10-15 03:15:16,764 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection147: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t176, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t176] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t176, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,635 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,238 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:16,764 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 176 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:28,664 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 253 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection143: begin an election at term 157 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,664 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection143: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t157, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t157] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t157, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:28,664 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 157 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,664 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,777 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_3 | 2019-10-15 03:15:28,690 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 267 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,777 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,288 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:7ms, electionTimeout:7ms datanode_2 | 2019-10-15 03:15:16,777 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 176 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,777 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:28,691 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,288 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 157 for changeToCandidate datanode_2 | 2019-10-15 03:15:16,791 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection148: begin an election at term 177 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,819 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 254 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:16,793 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 162 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:28,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,301 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection144: begin an election at term 158 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,317 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection144: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t158, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t158] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t158, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,819 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,793 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 255 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:16,807 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection148: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t177, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t177] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t177, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:16,317 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 158 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,807 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 177 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:28,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:16,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:16,393 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 173 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:28,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,872 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 163 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,393 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,863 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,872 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,393 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:16,394 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 268 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,872 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:16,423 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_3 | 2019-10-15 03:15:28,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,872 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:28,866 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,899 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:91ms, electionTimeout:91ms datanode_1 | 2019-10-15 03:15:16,423 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 158 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,866 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:16,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 177 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,907 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 269 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,437 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection145: begin an election at term 159 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:28,907 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,466 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection145: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t159, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t159] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t159, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:16,913 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection149: begin an election at term 178 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,907 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,466 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 159 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:16,929 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection149: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t178, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t178] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t178, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:28,907 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:16,929 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 178 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:28,961 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 256 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:16,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:28,962 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,594 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 174 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:16,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:28,962 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,594 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:16,984 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_1 | 2019-10-15 03:15:16,594 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:28,962 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,978 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 257 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,594 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,985 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 178 for changeToCandidate datanode_3 | 2019-10-15 03:15:28,978 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,622 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | 2019-10-15 03:15:28,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:16,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:16,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:28,979 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:16,998 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection150: begin an election at term 179 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:16,622 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 159 for changeToCandidate datanode_3 | 2019-10-15 03:15:29,023 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 258 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:17,015 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection150: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t179, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t179] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t179, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:16,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:29,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,015 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 179 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:16,651 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection146: begin an election at term 160 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:16,653 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 175 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:29,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,653 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,139 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 270 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,653 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,016 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 164 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:29,139 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,653 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,139 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,708 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection146: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t160, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,139 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,747 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection146: begin an election at term 161 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,016 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,231 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 259 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,749 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 176 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,102 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_3 | 2019-10-15 03:15:29,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 179 for changeToCandidate datanode_3 | 2019-10-15 03:15:29,231 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,749 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:29,284 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 271 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,764 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection146: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t161, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t161] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t161, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,116 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection151: begin an election at term 180 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,285 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 161 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:17,145 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection151: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t180, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t180] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t180, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,285 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:17,145 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 180 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:29,285 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,145 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:29,289 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 260 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,778 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | 2019-10-15 03:15:17,145 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,199 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 165 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:29,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 161 for changeToCandidate datanode_2 | 2019-10-15 03:15:17,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:29,289 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:16,791 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection147: begin an election at term 162 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,326 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 272 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,199 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,231 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | 2019-10-15 03:15:17,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,793 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 177 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,231 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 180 for changeToCandidate datanode_3 | 2019-10-15 03:15:29,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,245 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection152: begin an election at term 181 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:16,793 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:29,340 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 261 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:17,248 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 166 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,793 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:16,807 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection147: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t162, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t162] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t162, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,340 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,807 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 162 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:17,249 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,464 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 262 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:17,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection152: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t181, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t181] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t181, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,261 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 181 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:29,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,857 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_2 | 2019-10-15 03:15:17,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:29,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,511 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 273 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,857 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 162 for changeToCandidate datanode_2 | 2019-10-15 03:15:17,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 167 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:29,511 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:17,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,511 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,870 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection148: begin an election at term 163 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,511 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,886 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection148: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t163, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t163] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t163, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,541 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 274 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:16,886 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 163 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:17,399 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 168 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:29,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:17,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,541 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:16,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,915 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 178 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,399 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,647 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 263 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:16,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,417 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | 2019-10-15 03:15:29,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,915 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,417 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 181 for changeToCandidate datanode_1 | 2019-10-15 03:15:16,998 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_3 | 2019-10-15 03:15:29,647 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:16,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,702 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 275 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,445 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection153: begin an election at term 182 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:16,999 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 163 for changeToCandidate datanode_2 | 2019-10-15 03:15:17,462 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection153: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t182, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t182] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t182, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:16,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:17,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 182 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:29,703 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,000 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 179 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:29,799 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 264 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:17,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,580 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_1 | 2019-10-15 03:15:17,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:29,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,580 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,799 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,000 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,580 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 182 for changeToCandidate datanode_3 | 2019-10-15 03:15:29,833 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 276 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,014 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection149: begin an election at term 164 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,580 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:29,833 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,030 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection149: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t164, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t164] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t164, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:17,755 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection154: begin an election at term 183 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,833 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,030 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 164 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:17,758 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 169 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:29,833 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 265 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 180 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,758 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 277 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,787 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection154: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t183, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t183] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t183, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:29,988 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,119 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,787 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 183 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:29,988 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,787 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:17,183 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,787 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,898 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_1 | 2019-10-15 03:15:17,183 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 164 for changeToCandidate datanode_3 | 2019-10-15 03:15:30,051 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 266 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:17,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:30,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,197 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection150: begin an election at term 165 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 183 for changeToCandidate datanode_1 | 2019-10-15 03:15:17,213 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection150: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t165, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t165] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t165, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,051 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:17,213 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 165 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:17,912 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection155: begin an election at term 184 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:17,926 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection155: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t184, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t184] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t184, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:17,926 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 184 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,088 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 278 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,243 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | 2019-10-15 03:15:17,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,243 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,243 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 165 for changeToCandidate datanode_2 | 2019-10-15 03:15:17,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 170 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:30,088 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,243 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:17,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,112 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 267 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,246 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 181 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:30,112 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,112 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:30,112 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,928 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,247 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:30,239 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 279 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:17,976 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 171 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,247 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection151: begin an election at term 166 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:17,976 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection151: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t166, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t166] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t166, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:17,976 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 166 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,239 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:17,976 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 268 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,111 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_1 | 2019-10-15 03:15:17,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,266 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 280 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,111 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 184 for changeToCandidate datanode_1 | 2019-10-15 03:15:17,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:17,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 166 for changeToCandidate datanode_3 | 2019-10-15 03:15:30,292 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,116 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection156: begin an election at term 185 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,118 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 172 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,279 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection152: begin an election at term 167 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,118 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,286 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection152: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t167, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t167] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t167, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,292 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,286 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 167 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,333 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 281 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,119 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,286 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,333 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,286 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,135 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection156: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t185, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t185] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t185, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,333 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,383 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_2 | 2019-10-15 03:15:18,135 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 185 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,333 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,360 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 269 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,383 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 167 for changeToCandidate datanode_2 | 2019-10-15 03:15:18,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_3 | 2019-10-15 03:15:30,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection153: begin an election at term 168 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,360 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,446 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection153: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t168, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t168] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t168, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,153 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 185 for changeToCandidate datanode_3 | 2019-10-15 03:15:30,485 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 282 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 168 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:18,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:17,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:18,158 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection157: begin an election at term 186 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,160 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 173 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:30,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,447 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 182 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,447 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,160 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,485 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,447 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,160 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,448 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:30,536 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 270 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,174 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection157: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t186, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t186] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t186, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,636 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_2 | 2019-10-15 03:15:18,174 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 186 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,636 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 168 for changeToCandidate datanode_2 | 2019-10-15 03:15:18,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,536 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 174 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,755 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection154: begin an election at term 169 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,697 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 283 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 183 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:30,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,295 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 175 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:30,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,758 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,787 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection154: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t169, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t169] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t169, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 271 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:17,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 169 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:30,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,295 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:30,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,311 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 176 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:17,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:30,774 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,914 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 184 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,914 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,885 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 284 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:17,914 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,311 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,914 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:30,885 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_1 | 2019-10-15 03:15:17,920 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_3 | 2019-10-15 03:15:30,885 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,886 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,350 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 186 for changeToCandidate datanode_1 | 2019-10-15 03:15:17,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 169 for changeToCandidate datanode_3 | 2019-10-15 03:15:30,936 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 285 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:17,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,363 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection158: begin an election at term 187 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,925 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection155: begin an election at term 170 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,388 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection158: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t187, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t187] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t187, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:17,950 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection155: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t170, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,388 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 187 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:17,974 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection155: begin an election at term 171 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,937 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,389 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:17,990 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection155: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t171, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t171] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t171, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:30,992 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 272 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,389 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:30,992 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:17,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 171 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:18,389 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 177 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:30,992 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:17,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:18,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:30,992 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:17,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 273 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,020 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 286 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,390 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,020 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,020 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 171 for changeToCandidate datanode_2 | 2019-10-15 03:15:18,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 187 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,020 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:31,049 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,459 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection159: begin an election at term 188 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,116 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection156: begin an election at term 172 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,049 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,476 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection159: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t188, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t188] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t188, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,118 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 185 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:31,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 274 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,476 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 188 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,119 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:31,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection156: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t172, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t172] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t172, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,596 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 178 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:31,137 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 275 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,135 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 172 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:18,596 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,137 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,135 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:18,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:31,137 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,135 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,597 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:31,137 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,144 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:15:18,633 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_3 | 2019-10-15 03:15:31,169 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 287 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,144 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 172 for changeToCandidate datanode_2 | 2019-10-15 03:15:18,633 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 188 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:18,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:31,169 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,646 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection160: begin an election at term 189 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,158 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection157: begin an election at term 173 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 288 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,662 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection160: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t189, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t189] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t189, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,159 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 186 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:18,662 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 189 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:18,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:31,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,160 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,734 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:71ms, electionTimeout:71ms datanode_3 | 2019-10-15 03:15:31,355 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 276 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,174 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection157: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t173, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t173] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t173, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,355 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,174 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 173 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:18,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 189 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,355 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:18,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:31,356 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,748 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection161: begin an election at term 190 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,385 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 289 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,253 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_2 | 2019-10-15 03:15:18,765 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection161: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t190, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t190] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t190, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,385 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,253 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 190 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,385 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,253 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 173 for changeToCandidate datanode_2 | 2019-10-15 03:15:18,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,385 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,253 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:31,402 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 277 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,279 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: begin an election at term 174 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,803 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 179 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t174, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,294 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: begin an election at term 175 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,303 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t175, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,803 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,310 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: begin an election at term 176 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,402 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,824 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_3 | 2019-10-15 03:15:31,474 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 278 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,338 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection158: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t176, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t176] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t176, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:18,824 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,824 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 190 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,474 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,824 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:18,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 176 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,475 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,837 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection162: begin an election at term 191 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:18,845 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection162: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t191, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t191] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t191, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,475 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,846 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 191 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,504 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 279 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,364 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_2 | 2019-10-15 03:15:18,846 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,846 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,364 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 176 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:31,504 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:31,536 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 280 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:31,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,536 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:31,536 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,947 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_3 | 2019-10-15 03:15:31,576 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 290 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:18,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,576 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,365 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 187 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:31,576 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,576 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,947 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 191 for changeToCandidate datanode_1 | 2019-10-15 03:15:18,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:31,735 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 281 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:18,365 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:31,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,968 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection163: begin an election at term 192 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,387 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection159: begin an election at term 177 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:18,969 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 180 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,405 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection159: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t177, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t177] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t177, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,736 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:18,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,405 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 177 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,766 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 282 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:18,970 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:31,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:18,972 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection163: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t192, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,461 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 188 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection163: begin an election at term 193 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,794 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 291 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,030 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection163: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t193, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t193] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t193, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,462 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 193 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,462 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,794 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,580 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | 2019-10-15 03:15:19,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,873 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 283 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:19,048 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_1 | 2019-10-15 03:15:18,580 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 177 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:31,873 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:19,048 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 193 for changeToCandidate datanode_3 | 2019-10-15 03:15:31,873 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,594 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection160: begin an election at term 178 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:31,901 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 292 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection160: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t178, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t178] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t178, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,056 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection164: begin an election at term 194 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,612 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 178 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 181 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:31,901 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:31,955 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 293 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,648 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 189 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:31,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,059 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,075 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection164: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t194, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t194] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t194, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:31,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,075 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 194 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:31,956 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,648 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:32,060 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 284 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,750 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 190 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:32,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,142 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 182 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,750 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,061 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,750 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:32,061 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,751 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,157 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 294 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,142 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,778 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | 2019-10-15 03:15:19,192 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | 2019-10-15 03:15:32,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,157 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 178 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,192 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 194 for changeToCandidate datanode_3 | 2019-10-15 03:15:32,171 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 295 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:19,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:32,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,210 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection165: begin an election at term 195 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,801 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection161: begin an election at term 179 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,246 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection165: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t195, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t195] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t195, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection161: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t179, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t179] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t179, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,246 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 195 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:18,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 179 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:19,246 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:32,171 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:19,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:32,207 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 285 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:18,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,247 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 183 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:32,207 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,839 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 191 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,207 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:32,207 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,248 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,284 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 296 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,839 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,286 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:15:32,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,962 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_2 | 2019-10-15 03:15:19,286 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:18,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,286 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 195 for changeToCandidate datanode_3 | 2019-10-15 03:15:32,284 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:18,962 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 179 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,286 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:32,371 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 297 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:18,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:32,371 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:18,968 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection162: begin an election at term 180 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,292 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection166: begin an election at term 196 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,371 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,299 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection166: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t196, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t196] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t196, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:18,970 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 192 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:32,372 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,299 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 196 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,433 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 298 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:18,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:32,433 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:18,970 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,433 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,412 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 184 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,012 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection162: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t180, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t180] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t180, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,433 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,412 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,012 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 180 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:32,479 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 286 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:19,412 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,412 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,452 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | 2019-10-15 03:15:32,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,495 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 299 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,452 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 196 for changeToCandidate datanode_3 | 2019-10-15 03:15:32,495 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:32,495 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 193 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:32,495 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,493 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection167: begin an election at term 197 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,585 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 300 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,509 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection167: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t197, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t197] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t197, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:32,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,510 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 197 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:32,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,014 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,042 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | 2019-10-15 03:15:32,697 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 287 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,578 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 185 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:32,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,043 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 180 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,043 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:19,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:32,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection163: begin an election at term 181 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,579 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,721 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 288 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 194 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:32,721 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,601 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:91ms, electionTimeout:91ms datanode_1 | 2019-10-15 03:15:19,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,721 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:32,721 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,601 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 197 for changeToCandidate datanode_1 | 2019-10-15 03:15:19,059 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,794 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 301 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:19,075 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection163: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t181, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t181] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t181, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,615 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection168: begin an election at term 198 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,075 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 181 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:32,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,620 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection168: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t198, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:32,794 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,894 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 302 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,637 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection168: begin an election at term 199 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,125 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | 2019-10-15 03:15:19,639 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 186 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:32,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:32,895 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,126 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 181 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:32,936 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 289 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:19,639 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:32,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection164: begin an election at term 182 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,659 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection168: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t199, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t199] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t199, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:32,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,188 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection164: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t182, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t182] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t182, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 199 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:32,936 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,188 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 182 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:19,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,188 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:19,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:33,029 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 290 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:19,722 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_1 | 2019-10-15 03:15:19,188 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,029 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,212 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 195 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:33,029 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,722 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 199 for changeToCandidate datanode_1 | 2019-10-15 03:15:19,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,029 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:19,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,081 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 291 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:19,757 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection169: begin an election at term 200 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,212 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,771 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection169: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t200, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t200] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t200, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,224 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | 2019-10-15 03:15:33,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,771 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 200 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:33,082 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,224 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 182 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 292 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:19,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection165: begin an election at term 183 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,792 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 303 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 200 for changeToCandidate datanode_3 | 2019-10-15 03:15:33,097 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,256 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection165: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t183, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t183] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t183, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:19,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 183 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:19,800 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection170: begin an election at term 201 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,097 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:19,819 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection170: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t201, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t201] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t201, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:19,819 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 201 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:33,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 293 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,294 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 196 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:19,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,852 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 187 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,294 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,852 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,158 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 304 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,395 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_2 | 2019-10-15 03:15:19,852 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:33,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,852 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,890 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 188 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,395 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 183 for changeToCandidate datanode_2 | 2019-10-15 03:15:19,890 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:33,159 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,890 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,409 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection166: begin an election at term 184 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,180 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 294 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:19,890 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,418 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection166: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t184, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t184] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t184, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:19,987 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 189 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,418 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 184 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:19,418 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:19,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,418 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:19,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,496 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 197 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:33,180 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:19,987 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,209 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 305 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,009 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | 2019-10-15 03:15:33,209 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,009 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,209 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,009 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 201 for changeToCandidate datanode_3 | 2019-10-15 03:15:33,209 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,009 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:19,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,298 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 295 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:20,023 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection171: begin an election at term 202 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,496 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,038 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection171: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t202, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t202] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t202, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,564 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_3 | 2019-10-15 03:15:33,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,038 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 202 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:19,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,298 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,565 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 184 for changeToCandidate datanode_3 | 2019-10-15 03:15:33,373 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 306 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:33,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,217 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_3 | 2019-10-15 03:15:33,373 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,576 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection167: begin an election at term 185 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:20,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,373 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,604 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection167: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t185, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t185] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t185, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:20,218 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 202 for changeToCandidate datanode_3 | 2019-10-15 03:15:33,443 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 296 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 185 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:33,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:20,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:33,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,229 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 190 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:33,443 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,617 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 198 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:33,516 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 307 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,617 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,617 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,236 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection172: begin an election at term 203 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,618 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,272 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection172: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t203, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t203] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t203, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,627 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_3 | 2019-10-15 03:15:33,516 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,272 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 203 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:19,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 185 for changeToCandidate datanode_2 | 2019-10-15 03:15:20,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:33,657 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 297 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:20,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection168: begin an election at term 186 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 191 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,639 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 199 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:33,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,657 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,639 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:33,673 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 308 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,388 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,639 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,466 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_1 | 2019-10-15 03:15:19,659 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection168: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t186, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t186] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t186, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,673 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,659 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 186 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:20,466 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 203 for changeToCandidate datanode_3 | 2019-10-15 03:15:33,735 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 309 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:20,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:33,735 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,486 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection173: begin an election at term 204 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,735 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,759 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 200 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,512 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection173: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t204, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t204] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t204, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:33,735 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,512 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 204 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:33,866 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 298 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,512 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:33,866 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,759 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,866 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,512 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,801 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 201 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:33,866 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,589 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 192 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:33,953 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 310 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,802 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:33,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,802 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:33,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,802 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,590 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:33,953 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_2 | 2019-10-15 03:15:20,681 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_3 | 2019-10-15 03:15:34,017 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 299 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,017 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,842 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 186 for changeToCandidate datanode_2 | 2019-10-15 03:15:20,681 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 204 for changeToCandidate datanode_3 | 2019-10-15 03:15:34,017 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:34,018 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:19,850 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection169: begin an election at term 187 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,033 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 311 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:20,695 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection174: begin an election at term 205 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,711 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection174: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t205, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t205] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t205, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,860 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection169: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t187, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t187] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t187, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,711 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 205 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:19,860 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 187 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,033 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,711 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:19,860 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,169 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 300 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:20,711 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:19,860 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:34,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,806 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 193 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:34,169 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 312 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,873 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | 2019-10-15 03:15:20,806 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,170 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,806 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:34,170 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 187 for changeToCandidate datanode_2 | 2019-10-15 03:15:20,807 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,170 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:20,809 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_3 | 2019-10-15 03:15:34,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,887 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection170: begin an election at term 188 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,170 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,809 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 205 for changeToCandidate datanode_1 | 2019-10-15 03:15:19,910 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection170: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t188, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t188] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t188, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,253 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 301 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:20,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:19,910 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 188 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,813 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection175: begin an election at term 206 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:19,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,820 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection175: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t206, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t206] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t206, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,253 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,820 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 206 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,280 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 313 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:19,970 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | 2019-10-15 03:15:20,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:19,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:19,970 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 188 for changeToCandidate datanode_2 | 2019-10-15 03:15:20,944 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_3 | 2019-10-15 03:15:34,280 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:19,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:20,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,459 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 302 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:19,984 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection171: begin an election at term 189 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:20,945 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 206 for changeToCandidate datanode_1 | 2019-10-15 03:15:20,011 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection171: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t189, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t189] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t189, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:20,011 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 189 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:20,957 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection176: begin an election at term 207 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:20,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,459 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:20,965 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection176: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t207, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t207] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t207, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:20,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:20,965 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 207 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,498 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 314 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 202 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:34,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,965 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:20,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:20,965 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:34,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 194 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,024 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,498 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,201 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | 2019-10-15 03:15:34,696 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 303 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:34,696 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 189 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,006 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,696 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:34,696 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,226 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection172: begin an election at term 190 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:20,238 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 203 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:34,705 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 315 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,045 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_1 | 2019-10-15 03:15:20,238 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,705 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,045 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,238 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:34,705 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,046 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 207 for changeToCandidate datanode_1 | 2019-10-15 03:15:20,238 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,705 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,046 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:34,731 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 316 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,259 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection172: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t190, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t190] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t190, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,059 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection177: begin an election at term 208 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,731 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,259 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 190 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:21,076 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection177: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t208, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t208] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t208, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,731 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:21,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 208 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,731 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,799 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 317 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,370 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | 2019-10-15 03:15:34,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,092 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_1 | 2019-10-15 03:15:20,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 190 for changeToCandidate datanode_3 | 2019-10-15 03:15:34,799 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,093 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 208 for changeToCandidate datanode_1 | 2019-10-15 03:15:20,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection173: begin an election at term 191 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:20,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection173: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t191, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t191] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t191, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:34,839 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 304 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:21,106 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection178: begin an election at term 209 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:20,402 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 191 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:34,839 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 195 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,402 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:34,839 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,402 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,488 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 204 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:34,839 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,109 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,929 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 318 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,122 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection178: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t209, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t209] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t209, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:20,488 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 209 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:20,574 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:34,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:20,574 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,929 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,574 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 191 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,227 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 196 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,574 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,948 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 305 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,587 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection174: begin an election at term 192 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:34,948 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,604 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection174: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t192, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t192] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t192, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,227 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:34,948 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 192 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:21,291 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_3 | 2019-10-15 03:15:34,948 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:21,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:34,988 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 319 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 209 for changeToCandidate datanode_3 | 2019-10-15 03:15:34,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,696 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 205 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:34,988 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:34,988 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,298 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection179: begin an election at term 210 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,163 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 306 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,697 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,300 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 197 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:35,163 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,791 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_3 | 2019-10-15 03:15:35,163 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,163 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,791 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 192 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,301 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,187 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 320 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,315 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection179: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t210, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t210] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t210, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,187 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection175: begin an election at term 193 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 210 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:20,811 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection175: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t193, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t193] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t193, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,188 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,811 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 193 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:21,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:35,233 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 321 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,811 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:21,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:35,233 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,316 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 198 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:35,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,815 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 206 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,234 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:21,316 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,243 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 307 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,338 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_3 | 2019-10-15 03:15:35,243 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,815 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,244 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,959 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 207 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 210 for changeToCandidate datanode_3 | 2019-10-15 03:15:35,244 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:20,959 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:20,959 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,351 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection180: begin an election at term 211 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 322 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:20,959 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,367 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection180: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t211, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t211] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t211, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:20,981 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | 2019-10-15 03:15:21,368 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 211 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:35,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:20,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,368 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:35,367 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,368 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:20,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 193 for changeToCandidate datanode_3 | 2019-10-15 03:15:35,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 308 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:21,368 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 199 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:20,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:35,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,368 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,004 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection176: begin an election at term 194 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,020 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection176: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t194, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t194] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t194, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,369 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,020 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 194 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 323 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,469 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 200 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 309 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,061 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 208 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,469 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,640 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,061 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,098 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_3 | 2019-10-15 03:15:35,640 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,539 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:15:35,673 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 324 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 194 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,539 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,539 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 211 for changeToCandidate datanode_3 | 2019-10-15 03:15:35,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,106 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection177: begin an election at term 195 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,539 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:35,673 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,108 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 209 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,546 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection181: begin an election at term 212 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,828 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 310 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,548 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 201 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:35,828 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,108 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,108 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,828 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,122 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection177: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t195, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t195] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t195, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,828 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,548 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,122 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 195 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:35,841 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 311 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:21,564 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection181: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t212, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t212] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t212, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:35,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,564 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 212 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:21,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:35,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:21,211 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_2 | 2019-10-15 03:15:21,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,841 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,211 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 195 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,752 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:35,878 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 325 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,753 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 212 for changeToCandidate datanode_3 | 2019-10-15 03:15:35,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:35,878 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,225 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection178: begin an election at term 196 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:35,913 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 312 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,243 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection178: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t196, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t196] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t196, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,757 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection182: begin an election at term 213 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,913 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,913 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,759 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 202 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,243 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 196 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:35,913 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,243 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:21,759 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:35,972 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 326 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,243 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,759 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:35,972 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,283 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_2 | 2019-10-15 03:15:21,774 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection182: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t213, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t213] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t213, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:35,972 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,774 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 213 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:35,972 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,284 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 196 for changeToCandidate datanode_2 | 2019-10-15 03:15:21,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:36,014 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 327 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:21,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,298 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection179: begin an election at term 197 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,814 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:15:36,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,300 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 210 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:36,014 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,097 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 313 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 213 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,301 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:36,097 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:21,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection183: begin an election at term 214 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,303 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection179: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t197, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,097 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,844 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection183: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t214, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t214] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t214, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection179: begin an election at term 198 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,844 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 214 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:36,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 328 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,330 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection179: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t198, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t198] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t198, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:21,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:36,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,330 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 198 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:21,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:21,885 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 203 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:36,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,331 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,885 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,344 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 314 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:21,885 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,344 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,353 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 211 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:21,885 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:36,345 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,353 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:21,987 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:15:36,345 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,353 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:21,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,347 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 329 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,353 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:21,987 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 214 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,347 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,361 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | 2019-10-15 03:15:21,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:36,347 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,347 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 198 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,384 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 315 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:22,001 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection184: begin an election at term 215 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,384 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,040 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection184: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t215, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t215] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t215, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,384 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,040 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 215 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:21,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:36,384 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:21,367 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection180: begin an election at term 199 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,526 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 330 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,373 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection180: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t199, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t199] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t199, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,044 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 204 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:36,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,373 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 199 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,044 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,526 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,044 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 316 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,044 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 331 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,453 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:15:22,192 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,192 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 215 for changeToCandidate datanode_1 | 2019-10-15 03:15:21,453 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 199 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,453 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:21,466 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection181: begin an election at term 200 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,611 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection185: begin an election at term 216 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,611 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,232 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection185: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t216, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t216] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t216, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,655 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 317 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:22,232 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 216 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:36,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection181: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t200, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t200] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t200, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:36,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,484 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 200 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 205 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:36,655 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,801 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 318 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,532 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_3 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,234 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:36,801 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,269 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | 2019-10-15 03:15:36,801 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 332 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,269 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,532 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 200 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:36,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,270 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 216 for changeToCandidate datanode_1 | 2019-10-15 03:15:21,546 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection182: begin an election at term 201 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:36,802 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:21,548 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 212 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:36,820 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 333 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,289 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection186: begin an election at term 217 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,820 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,303 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection186: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t217, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t217] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t217, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:36,820 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 217 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:21,548 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:21,564 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection182: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t201, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t201] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t201, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,820 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,564 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 201 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 206 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:36,964 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 319 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:36,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:36,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,741 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:15:22,339 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:36,964 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,742 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,446 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_1 | 2019-10-15 03:15:21,742 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 201 for changeToCandidate datanode_3 | 2019-10-15 03:15:36,996 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 334 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,742 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:36,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,446 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 217 for changeToCandidate datanode_1 | 2019-10-15 03:15:21,757 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection183: begin an election at term 202 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:21,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 213 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:36,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,450 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection187: begin an election at term 218 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,450 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 207 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:36,996 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,450 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,084 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 335 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,759 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,450 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:37,084 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,774 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection183: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t202, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t202] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t202, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,450 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,084 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 202 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,465 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection187: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t218, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t218] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t218, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,466 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 218 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:37,084 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:37,099 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 320 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,830 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 214 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:37,099 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,830 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,578 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 208 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:37,099 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,830 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,099 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:21,830 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:21,870 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | 2019-10-15 03:15:22,578 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,264 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 336 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:21,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | 2019-10-15 03:15:37,264 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:21,870 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 202 for changeToCandidate datanode_2 | 2019-10-15 03:15:22,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,265 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:22,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 218 for changeToCandidate datanode_1 | 2019-10-15 03:15:21,883 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection184: begin an election at term 203 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,265 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:37,275 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 321 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:21,899 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection184: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t203, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t203] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t203, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,275 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,275 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:21,900 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 203 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:37,275 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection188: begin an election at term 219 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,900 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:37,354 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 322 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:22,631 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection188: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t219, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t219] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t219, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:21,900 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,631 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 219 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:22,003 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 215 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:37,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,631 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,354 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,631 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 337 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,807 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 209 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:37,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,004 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,024 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | 2019-10-15 03:15:22,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:37,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,808 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 323 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 203 for changeToCandidate datanode_2 | 2019-10-15 03:15:22,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:22,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 338 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:22,042 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection185: begin an election at term 204 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 219 for changeToCandidate datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,067 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection185: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t204, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t204] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t204, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,067 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 204 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,821 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection189: begin an election at term 220 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,439 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection189: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t220, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:22,841 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection189: begin an election at term 221 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,439 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 216 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection189: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t221, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t221] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t221, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,520 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 339 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 221 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:22,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,520 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,520 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,219 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,520 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,896 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_3 | 2019-10-15 03:15:37,590 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 340 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:22,896 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,226 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_3 | 2019-10-15 03:15:37,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:22,896 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 221 for changeToCandidate datanode_1 | 2019-10-15 03:15:22,226 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,590 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:22,897 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,226 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 204 for changeToCandidate datanode_3 | 2019-10-15 03:15:37,590 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:22,910 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection190: begin an election at term 222 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,226 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:37,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 324 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,232 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection186: begin an election at term 205 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:22,927 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection190: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t222, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t222] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t222, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,250 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection186: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t205, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t205] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t205, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,251 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 205 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:22,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 222 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:37,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:22,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,732 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 325 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:22,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,291 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 217 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:37,732 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 210 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,732 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,732 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,292 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,007 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,765 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 326 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,325 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | 2019-10-15 03:15:23,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_3 | 2019-10-15 03:15:37,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,325 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 205 for changeToCandidate datanode_2 | 2019-10-15 03:15:23,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 222 for changeToCandidate datanode_3 | 2019-10-15 03:15:37,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,032 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:37,831 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 341 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,044 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection191: begin an election at term 223 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection187: begin an election at term 206 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,831 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,061 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection191: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t223, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t223] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t223, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,831 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,061 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 223 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:22,346 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection187: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t206, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t206] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t206, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,831 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,061 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,346 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 206 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:37,849 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 327 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:23,061 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:37,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,111 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_1 | 2019-10-15 03:15:22,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,419 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_3 | 2019-10-15 03:15:37,849 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 223 for changeToCandidate datanode_1 | 2019-10-15 03:15:22,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,899 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 342 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,419 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 206 for changeToCandidate datanode_3 | 2019-10-15 03:15:37,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,113 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 211 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:37,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,447 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection188: begin an election at term 207 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,899 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:37,938 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 328 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:23,113 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,452 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 218 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:37,938 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,152 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection192: begin an election at term 224 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:37,938 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,178 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection192: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t224, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t224] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t224, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:37,938 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,179 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 224 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:22,455 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:37,965 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 329 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:23,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,465 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection188: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t207, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t207] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t207, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:37,965 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,466 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 207 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:23,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:37,965 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:23,254 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 212 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:37,965 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,254 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,027 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 343 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:22,561 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:94ms, electionTimeout:94ms datanode_2 | 2019-10-15 03:15:23,254 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,027 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,254 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,027 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 207 for changeToCandidate datanode_2 | 2019-10-15 03:15:23,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | 2019-10-15 03:15:38,027 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:23,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,064 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 344 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,290 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 224 for changeToCandidate datanode_1 | 2019-10-15 03:15:22,576 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection189: begin an election at term 208 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection189: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t208, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t208] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t208, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,304 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection193: begin an election at term 225 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,612 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 208 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,064 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,320 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection193: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t225, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t225] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t225, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,087 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 345 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,320 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 225 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:22,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:38,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:22,613 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 219 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:38,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,087 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,320 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:38,184 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 330 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:23,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_1 | 2019-10-15 03:15:22,613 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,791 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_3 | 2019-10-15 03:15:38,185 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:38,185 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,440 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 225 for changeToCandidate datanode_1 | 2019-10-15 03:15:22,791 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 208 for changeToCandidate datanode_3 | 2019-10-15 03:15:38,262 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 346 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:23,444 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection194: begin an election at term 226 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,262 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,805 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection190: begin an election at term 209 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:23,444 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 213 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:38,262 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,822 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection190: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t209, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t209] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t209, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:23,444 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,822 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 209 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:23,444 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,262 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,444 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,332 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 331 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:23,452 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection194: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t226, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t226] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t226, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,822 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:23,452 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 226 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,822 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,332 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,823 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 220 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:22,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,398 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 347 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,503 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_1 | 2019-10-15 03:15:22,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,823 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,503 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 226 for changeToCandidate datanode_1 | 2019-10-15 03:15:22,842 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 221 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:38,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:22,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 348 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,515 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection195: begin an election at term 227 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:22,843 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,536 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection195: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t227, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t227] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t227, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,912 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 222 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,536 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 227 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,912 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,514 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,912 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 332 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:22,912 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,539 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 214 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:22,990 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | 2019-10-15 03:15:23,539 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:22,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,539 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,514 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:22,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 209 for changeToCandidate datanode_2 | 2019-10-15 03:15:23,539 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,542 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 349 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:22,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:23,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | 2019-10-15 03:15:38,542 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,003 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection191: begin an election at term 210 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:23,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection191: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t210, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t210] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t210, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:23,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 227 for changeToCandidate datanode_3 | 2019-10-15 03:15:38,542 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:23,021 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 210 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,542 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,702 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection196: begin an election at term 228 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 350 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,707 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 215 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 223 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,707 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 333 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,708 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection196: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t228, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t228] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t228, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,047 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,709 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 228 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,657 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,098 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_2 | 2019-10-15 03:15:23,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 210 for changeToCandidate datanode_2 | 2019-10-15 03:15:23,848 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 216 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:38,657 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:23,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,720 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 334 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,111 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection192: begin an election at term 211 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,720 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,153 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection192: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t211, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t211] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t211, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,720 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,153 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 211 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:23,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,153 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,720 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,848 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,153 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:38,778 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 351 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,873 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:15:23,154 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 224 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:38,778 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,873 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,154 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,778 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,873 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 228 for changeToCandidate datanode_1 | 2019-10-15 03:15:23,154 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:23,874 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:23,154 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,879 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection197: begin an election at term 229 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,778 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,237 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:83ms, electionTimeout:83ms datanode_2 | 2019-10-15 03:15:23,881 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 217 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:38,826 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 335 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:38,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,237 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 211 for changeToCandidate datanode_2 | 2019-10-15 03:15:23,881 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:38,826 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:23,895 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection197: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t229, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t229] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t229, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,956 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 352 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:23,895 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 229 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:23,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:38,956 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:23,895 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:23,251 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection193: begin an election at term 212 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,956 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:23,895 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,268 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection193: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t212, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t212] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t212, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:38,956 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,000 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 218 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,268 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 212 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:38,990 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 336 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,000 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,268 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:38,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,000 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,268 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,000 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,305 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 225 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,015 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 219 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:38,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:38,990 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,015 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,306 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,103 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 337 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,024 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_1 | 2019-10-15 03:15:23,428 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_3 | 2019-10-15 03:15:39,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,429 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 212 for changeToCandidate datanode_2 | 2019-10-15 03:15:24,024 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 229 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,103 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:24,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:39,118 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 353 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,442 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection194: begin an election at term 213 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,039 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection198: begin an election at term 230 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 226 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,056 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection198: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t230, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t230] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t230, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,057 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 230 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:23,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,450 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection194: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t213, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t213] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t213, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 338 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_1 | 2019-10-15 03:15:23,450 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 213 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:24,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 230 for changeToCandidate datanode_1 | 2019-10-15 03:15:23,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:23,519 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 227 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,118 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection199: begin an election at term 231 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,124 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection199: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t231, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t231] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t231, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,189 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 339 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 231 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,189 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:23,519 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,189 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,525 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | 2019-10-15 03:15:39,189 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,157 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 220 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:39,234 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 354 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,525 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,525 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 213 for changeToCandidate datanode_2 | 2019-10-15 03:15:24,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,525 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:24,157 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,234 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection195: begin an election at term 214 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,318 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | 2019-10-15 03:15:39,307 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 340 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,547 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection195: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t214, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t214] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t214, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,547 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 214 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:24,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 231 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:23,547 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,307 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,344 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection200: begin an election at term 232 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,394 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 355 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,547 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,348 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 221 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:39,394 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,690 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_2 | 2019-10-15 03:15:24,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,395 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,348 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,395 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,691 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 214 for changeToCandidate datanode_2 | 2019-10-15 03:15:24,361 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection200: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t232, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t232] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t232, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,491 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 341 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,691 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:24,361 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 232 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,491 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,704 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 228 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,361 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,491 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,361 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,491 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,479 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | 2019-10-15 03:15:39,529 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 356 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,704 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,529 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,705 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection196: begin an election at term 215 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,479 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 232 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,529 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,712 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection196: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t215, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t215] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t215, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:39,529 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,712 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 215 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:24,507 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection201: begin an election at term 233 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,571 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 342 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,539 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection201: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t233, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t233] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t233, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,712 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,571 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:39,571 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,832 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | 2019-10-15 03:15:24,539 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 233 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,572 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,539 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,614 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 357 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,832 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 215 for changeToCandidate datanode_2 | 2019-10-15 03:15:24,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,542 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 222 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:39,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection197: begin an election at term 216 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,614 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,862 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection197: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t216, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t216] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t216, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,765 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 358 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:23,862 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 216 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:24,542 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:24,567 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_3 | 2019-10-15 03:15:39,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,765 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,567 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 233 for changeToCandidate datanode_1 | 2019-10-15 03:15:23,865 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_3 | 2019-10-15 03:15:39,797 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 343 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:39,797 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,865 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 216 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,798 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,580 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection202: begin an election at term 234 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:24,597 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection202: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t234, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t234] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t234, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,879 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection198: begin an election at term 217 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,597 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 234 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,798 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,881 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 229 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,826 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 344 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:39,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,683 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 223 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:39,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:23,881 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,683 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,826 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,895 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection198: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t217, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t217] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t217, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,683 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:23,895 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 217 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:23,895 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,883 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 359 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,683 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:23,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:24,718 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_1 | 2019-10-15 03:15:23,963 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | 2019-10-15 03:15:24,718 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,883 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:23,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,718 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 234 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,883 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,718 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:39,883 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,732 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection203: begin an election at term 235 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,963 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 217 for changeToCandidate datanode_3 | 2019-10-15 03:15:39,916 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 360 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,743 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection203: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t235, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t235] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t235, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:23,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:39,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,743 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 235 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:23,997 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection199: begin an election at term 218 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,743 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:24,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection199: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t218, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,916 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,743 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection199: begin an election at term 219 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,931 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 361 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 224 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,040 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection199: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t219, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t219] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t219, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:39,931 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,041 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 219 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:39,931 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:39,931 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:39,984 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 345 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,771 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,041 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 230 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:39,984 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,850 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_1 | 2019-10-15 03:15:24,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:39,984 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:39,984 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,042 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:39,999 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 346 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,851 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 235 for changeToCandidate datanode_1 | 2019-10-15 03:15:24,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 231 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:39,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:24,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,000 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,874 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection204: begin an election at term 236 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,000 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,889 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection204: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t236, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t236] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t236, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,120 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,150 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 362 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 236 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:24,151 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_3 | 2019-10-15 03:15:40,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:24,151 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,151 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 219 for changeToCandidate datanode_3 | 2019-10-15 03:15:40,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,890 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 225 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,151 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:40,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 347 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,155 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection200: begin an election at term 220 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:24,890 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,890 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,162 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection200: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t220, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t220] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t220, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,890 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,162 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 220 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:40,208 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_1 | 2019-10-15 03:15:24,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:40,302 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 363 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,302 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 236 for changeToCandidate datanode_1 | 2019-10-15 03:15:24,342 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_3 | 2019-10-15 03:15:40,302 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:24,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,302 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,910 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: begin an election at term 237 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 220 for changeToCandidate datanode_2 | 2019-10-15 03:15:24,922 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t237, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:40,333 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 364 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:24,942 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: begin an election at term 238 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,346 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 232 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:40,333 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,948 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t238, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,976 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: begin an election at term 239 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,333 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 348 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:24,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 226 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,346 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:24,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,346 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection201: begin an election at term 221 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,334 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:24,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,361 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection201: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t221, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t221] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t221, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:24,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 221 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,002 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection205: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t239, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t239] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t239, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,002 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 239 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:40,334 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:40,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 365 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:24,509 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 233 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:40,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,152 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 227 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,509 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,510 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,527 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 366 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,531 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_3 | 2019-10-15 03:15:40,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,174 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_1 | 2019-10-15 03:15:24,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,528 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,528 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,531 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 221 for changeToCandidate datanode_2 | 2019-10-15 03:15:25,175 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 239 for changeToCandidate datanode_1 | 2019-10-15 03:15:24,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:40,529 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 349 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:25,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:24,540 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection202: begin an election at term 222 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,182 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection206: begin an election at term 240 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,557 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection202: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t222, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t222] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t222, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,201 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection206: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t240, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t240] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t240, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,529 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 222 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,201 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 240 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:40,530 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:40,530 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:40,551 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 350 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,582 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 234 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,281 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_3 | 2019-10-15 03:15:40,551 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,552 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 240 for changeToCandidate datanode_1 | 2019-10-15 03:15:24,583 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,552 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:24,675 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 228 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,565 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 351 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,675 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 222 for changeToCandidate datanode_3 | 2019-10-15 03:15:40,565 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,676 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:24,680 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection203: begin an election at term 223 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,565 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,694 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection203: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t223, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t223] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t223, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,565 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,694 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 223 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:40,713 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 367 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,713 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,295 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection207: begin an election at term 241 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,734 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 235 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:40,713 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 352 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:25,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection207: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t241, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t241] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t241, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,312 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 241 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:24,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:40,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:24,734 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,761 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:25,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:40,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,458 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 229 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:40,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,761 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 223 for changeToCandidate datanode_2 | 2019-10-15 03:15:25,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,898 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 368 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:24,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:40,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,767 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection204: begin an election at term 224 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,459 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:40,898 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,467 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_1 | 2019-10-15 03:15:24,782 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection204: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t224, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t224] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t224, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,917 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 353 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:25,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:40,917 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 224 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,467 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 241 for changeToCandidate datanode_3 | 2019-10-15 03:15:40,917 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:40,918 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,484 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection208: begin an election at term 242 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,964 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 354 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,873 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:91ms, electionTimeout:91ms datanode_2 | 2019-10-15 03:15:25,499 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection208: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t242, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t242] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t242, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:40,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 242 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:40,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,874 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 224 for changeToCandidate datanode_2 | 2019-10-15 03:15:25,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:40,964 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:25,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,058 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 355 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,876 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 236 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,540 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_3 | 2019-10-15 03:15:41,058 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,541 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,058 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,541 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 242 for changeToCandidate datanode_3 | 2019-10-15 03:15:41,058 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,876 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,541 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:41,128 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 369 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:24,888 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection205: begin an election at term 225 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,555 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection209: begin an election at term 243 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:41,128 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,911 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection205: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t225, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t225] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t225, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,556 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 230 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:41,128 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,911 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 225 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,128 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,239 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 356 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:25,557 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,584 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection209: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t243, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t243] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t243, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,912 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 237 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:41,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,584 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 243 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:41,239 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:24,912 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,309 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 357 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:25,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,912 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,309 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 231 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:24,912 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,309 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,943 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 238 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:41,309 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,620 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:24,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,327 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 370 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,620 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:24,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,327 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,943 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,661 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_3 | 2019-10-15 03:15:41,327 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:24,965 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | 2019-10-15 03:15:25,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,328 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,661 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 243 for changeToCandidate datanode_1 | 2019-10-15 03:15:24,965 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 225 for changeToCandidate datanode_3 | 2019-10-15 03:15:41,388 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 371 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:24,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:41,388 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection210: begin an election at term 244 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,978 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 239 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:41,388 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:25,691 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection210: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t244, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t244] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t244, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:24,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,388 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,692 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 244 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:24,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,412 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 372 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:24,978 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:41,412 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:24,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection206: begin an election at term 226 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,412 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,002 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection206: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t226, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t226] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t226, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,855 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 232 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:41,412 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,002 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 226 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,423 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 358 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,423 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,855 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,423 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,136 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | 2019-10-15 03:15:25,879 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_1 | 2019-10-15 03:15:25,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,423 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,137 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 226 for changeToCandidate datanode_2 | 2019-10-15 03:15:25,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 244 for changeToCandidate datanode_3 | 2019-10-15 03:15:41,456 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 359 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:25,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:41,456 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,150 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection207: begin an election at term 227 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,903 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection211: begin an election at term 245 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:41,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,183 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection207: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t227, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t227] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t227, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:25,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection211: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t245, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t245] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t245, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:41,457 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 227 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:25,919 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 245 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:25,919 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,467 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 373 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:25,919 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 240 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:41,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,942 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 233 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:25,942 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:25,942 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,467 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,185 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:25,942 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,566 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 374 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:25,265 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:81ms, electionTimeout:81ms datanode_2 | 2019-10-15 03:15:26,093 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 234 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:41,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,265 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,265 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 227 for changeToCandidate datanode_2 | 2019-10-15 03:15:26,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,566 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,265 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:26,093 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,651 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 360 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection208: begin an election at term 228 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,117 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_3 | 2019-10-15 03:15:41,651 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection208: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t228, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t228] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t228, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,117 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,651 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,117 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 245 for changeToCandidate datanode_1 | 2019-10-15 03:15:25,296 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 228 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:41,651 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,118 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:25,296 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:41,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 375 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,131 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection212: begin an election at term 246 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,296 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,158 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection212: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t246, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t246] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t246, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,297 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 241 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:41,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,677 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 246 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:25,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,703 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 361 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:26,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:25,297 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:25,443 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_3 | 2019-10-15 03:15:41,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,248 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 235 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,443 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,703 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,443 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 228 for changeToCandidate datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 376 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:25,443 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 362 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,248 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,258 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | 2019-10-15 03:15:26,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:41,878 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,456 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection209: begin an election at term 229 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,258 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 246 for changeToCandidate datanode_3 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection209: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t229, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t229] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t229, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:41,878 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 229 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:26,262 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection213: begin an election at term 247 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:41,958 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 377 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,267 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection213: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t247, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:41,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,277 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection213: begin an election at term 248 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,293 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection213: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t248, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t248] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t248, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 242 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,293 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 248 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:25,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:41,958 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:25,485 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,355 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_3 | 2019-10-15 03:15:42,023 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 363 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,551 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:26,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,356 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 248 for changeToCandidate datanode_1 | 2019-10-15 03:15:25,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:25,551 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 229 for changeToCandidate datanode_3 | 2019-10-15 03:15:42,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,369 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection214: begin an election at term 249 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 364 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:26,371 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 236 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 378 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,371 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,555 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection210: begin an election at term 230 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,382 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection214: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t249, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,087 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,385 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection214: begin an election at term 250 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,557 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 243 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 237 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,087 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,557 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,158 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 365 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,557 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,584 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection210: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t230, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t230] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t230, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:42,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,584 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 230 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:42,159 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,397 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection214: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t250, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t250] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t250, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:42,268 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 379 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 250 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:25,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:42,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,604 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_2 | 2019-10-15 03:15:26,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:42,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,398 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,268 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 230 for changeToCandidate datanode_3 | 2019-10-15 03:15:42,353 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 366 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:26,576 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_1 | 2019-10-15 03:15:25,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:42,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,618 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection211: begin an election at term 231 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,576 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 250 for changeToCandidate datanode_1 | 2019-10-15 03:15:25,633 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection211: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t231, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t231] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t231, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,353 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:42,441 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 380 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:25,633 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 231 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:26,579 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 238 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:42,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:26,579 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:26,579 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,441 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 244 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,579 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:42,527 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 367 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,591 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection215: begin an election at term 251 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:26,608 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection215: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t251, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t251] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t251, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,677 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,608 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 251 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:42,527 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,786 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:15:26,608 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:42,636 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 381 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:25,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,608 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,636 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,786 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 231 for changeToCandidate datanode_2 | 2019-10-15 03:15:26,716 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 239 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:42,636 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,853 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection212: begin an election at term 232 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,636 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:25,869 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection212: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t232, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t232] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t232, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,701 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 368 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:25,870 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 232 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:26,717 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:42,701 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,806 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_1 | 2019-10-15 03:15:25,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:42,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,806 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:25,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:42,702 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:26,806 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 251 for changeToCandidate datanode_1 | 2019-10-15 03:15:25,905 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 245 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 382 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:26,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:25,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,830 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection216: begin an election at term 252 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 369 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:26,846 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection216: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t252, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,905 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:26,851 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection216: begin an election at term 253 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:25,929 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,854 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 240 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:42,813 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:26,854 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:25,929 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 232 for changeToCandidate datanode_2 | 2019-10-15 03:15:26,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,813 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:26,855 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:25,941 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection213: begin an election at term 233 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection216: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t253, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t253] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t253, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:42,952 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 383 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:26,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection213: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t233, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t233] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t233, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:26,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 253 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:42,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,016 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 233 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:26,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:42,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:26,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,952 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:26,970 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 241 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:42,978 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 370 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,065 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_2 | 2019-10-15 03:15:26,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 233 for changeToCandidate datanode_2 | 2019-10-15 03:15:26,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:42,978 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:26,971 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,091 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection214: begin an election at term 234 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,050 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 384 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,042 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_1 | 2019-10-15 03:15:26,107 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection214: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t234, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t234] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t234, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,107 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 234 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:43,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,042 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 253 for changeToCandidate datanode_1 | 2019-10-15 03:15:26,107 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,050 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,107 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,043 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:43,105 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 385 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:26,132 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 246 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,088 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection217: begin an election at term 254 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,105 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,104 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection217: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t254, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t254] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t254, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,105 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 254 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:43,105 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:26,132 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 371 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,232 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,115 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_1 | 2019-10-15 03:15:26,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 386 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 234 for changeToCandidate datanode_2 | 2019-10-15 03:15:27,115 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 254 for changeToCandidate datanode_3 | 2019-10-15 03:15:43,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:27,115 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection215: begin an election at term 235 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,141 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection218: begin an election at term 255 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,263 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection215: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t235, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t235] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t235, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,150 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection218: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t255, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,170 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection218: begin an election at term 256 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 235 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:43,259 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 387 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,192 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection218: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t256, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t256] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t256, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,259 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,192 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 256 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:26,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,259 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,259 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,264 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 247 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:43,274 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 372 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 242 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:43,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,264 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:43,274 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,278 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 248 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,193 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,309 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 388 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:26,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 243 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:43,309 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,309 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,279 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,309 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,381 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 373 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:131ms datanode_3 | 2019-10-15 03:15:43,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,354 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_2 | 2019-10-15 03:15:27,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,354 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 256 for changeToCandidate datanode_3 | 2019-10-15 03:15:43,381 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,354 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 235 for changeToCandidate datanode_2 | 2019-10-15 03:15:27,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:43,441 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 374 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,354 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:27,357 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection219: begin an election at term 257 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,369 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection216: begin an election at term 236 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,374 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection219: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t257, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t257] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t257, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 249 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:26,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,374 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 257 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:26,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,441 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:26,370 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,458 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 389 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,380 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection216: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t236, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,458 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,413 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 244 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection216: begin an election at term 237 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,458 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,386 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 250 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:43,458 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,386 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,550 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 375 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,414 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,387 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,453 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_1 | 2019-10-15 03:15:26,387 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,397 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection216: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t237, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t237] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t237, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,550 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,453 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 257 for changeToCandidate datanode_1 | 2019-10-15 03:15:26,397 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 237 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:27,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:26,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,638 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 376 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection220: begin an election at term 258 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,638 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,494 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection220: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t258, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t258] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t258, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,551 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | 2019-10-15 03:15:43,638 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,494 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 258 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:26,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,638 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:26,551 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 237 for changeToCandidate datanode_3 | 2019-10-15 03:15:43,655 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 390 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:43,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,540 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_1 | 2019-10-15 03:15:26,577 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection217: begin an election at term 238 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,592 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection217: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t238, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t238] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t238, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,655 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,540 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 258 for changeToCandidate datanode_1 | 2019-10-15 03:15:26,592 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 238 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 391 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,541 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:26,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:27,555 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection221: begin an election at term 259 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,583 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection221: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t259, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t259] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t259, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,593 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 251 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 377 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 259 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:26,593 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,593 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,853 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,596 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 245 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,593 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,596 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,702 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_3 | 2019-10-15 03:15:43,853 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,596 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 392 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,597 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,702 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 238 for changeToCandidate datanode_3 | 2019-10-15 03:15:43,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 246 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:43,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:27,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,612 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:43,929 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 378 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,612 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:43,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,737 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_3 | 2019-10-15 03:15:43,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,714 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection218: begin an election at term 239 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:43,929 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,737 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:43,975 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 379 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,743 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection218: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t239, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t239] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t239, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,737 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 259 for changeToCandidate datanode_1 | 2019-10-15 03:15:26,743 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 239 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:43,975 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,737 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:26,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:43,975 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,751 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection222: begin an election at term 260 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:43,975 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,832 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 252 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,779 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection222: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t260, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t260] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t260, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,125 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 380 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,779 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 260 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:44,125 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:26,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,125 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 393 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:27,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,126 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 247 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:44,126 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,849 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_2 | 2019-10-15 03:15:27,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,126 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:44,126 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,850 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 239 for changeToCandidate datanode_2 | 2019-10-15 03:15:27,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,126 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,842 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_1 | 2019-10-15 03:15:26,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:44,184 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 381 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:27,842 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,852 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 253 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:44,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,842 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 260 for changeToCandidate datanode_1 | 2019-10-15 03:15:26,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:27,842 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:26,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,184 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,855 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection223: begin an election at term 261 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:26,853 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection219: begin an election at term 240 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,884 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection223: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t261, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t261] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t261, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,200 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 394 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:26,853 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,884 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 261 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:44,200 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,861 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection219: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t240, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t240] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t240, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,884 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:44,200 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,861 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 240 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:44,200 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,861 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:26,861 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,374 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 382 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:26,942 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:81ms, electionTimeout:81ms datanode_1 | 2019-10-15 03:15:26,942 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,884 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,942 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 240 for changeToCandidate datanode_2 | 2019-10-15 03:15:27,904 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 248 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:44,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:27,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:26,968 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection220: begin an election at term 241 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:26,995 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection220: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t241, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t241] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t241, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:27,904 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:26,995 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 241 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:27,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_1 | 2019-10-15 03:15:26,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:44,374 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:27,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:26,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,981 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 261 for changeToCandidate datanode_1 | 2019-10-15 03:15:27,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 254 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:44,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 395 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:44,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:27,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:44,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:27,993 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection224: begin an election at term 262 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,420 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 383 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,090 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,008 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection224: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t262, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t262] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t262, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:27,143 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 255 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,008 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 262 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:27,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,008 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:44,420 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,008 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,546 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 384 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:28,052 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 249 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,143 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,546 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,546 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,163 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | 2019-10-15 03:15:28,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:44,546 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,052 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,599 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 396 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,163 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 241 for changeToCandidate datanode_2 | 2019-10-15 03:15:28,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_3 | 2019-10-15 03:15:44,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:28,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,171 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 256 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,195 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 262 for changeToCandidate datanode_3 | 2019-10-15 03:15:44,600 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,171 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,195 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:44,643 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 385 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,171 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:28,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection225: begin an election at term 263 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,643 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,171 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,245 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 250 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:44,643 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,191 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection221: begin an election at term 242 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,245 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,643 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,214 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection221: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t242, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t242] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t242, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,661 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 397 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,245 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,214 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 242 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:44,661 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,245 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,214 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:44,661 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,214 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:28,267 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection225: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t263, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t263] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t263, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,661 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,267 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 263 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:27,266 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_3 | 2019-10-15 03:15:44,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 386 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:28,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:27,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 242 for changeToCandidate datanode_3 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,271 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_1 | 2019-10-15 03:15:27,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:44,863 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,271 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,279 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection222: begin an election at term 243 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 398 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,271 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 263 for changeToCandidate datanode_1 | 2019-10-15 03:15:27,358 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection222: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t243, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t243] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t243, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,359 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 243 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:28,271 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:44,864 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:44,864 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,298 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection226: begin an election at term 264 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:27,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,906 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 387 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:28,303 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection226: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t264, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:27,360 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 257 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:44,906 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,313 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection226: begin an election at term 265 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:27,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,906 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,341 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection226: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t265, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t265] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t265, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:27,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:44,906 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,341 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 265 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:27,360 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,946 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 399 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,342 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:27,397 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | 2019-10-15 03:15:44,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,342 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,947 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 251 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,397 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 243 for changeToCandidate datanode_3 | 2019-10-15 03:15:44,947 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:44,973 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 400 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,411 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection223: begin an election at term 244 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,423 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:44,973 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,433 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection223: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t244, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t244] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t244, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,480 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | 2019-10-15 03:15:44,973 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 244 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:28,480 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:44,973 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:28,480 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 265 for changeToCandidate datanode_3 | 2019-10-15 03:15:45,026 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 388 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:28,481 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:45,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,478 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 258 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,495 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection227: begin an election at term 266 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,527 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection227: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t266, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t266] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t266, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,026 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:28,527 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 266 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:27,478 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,527 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:45,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 401 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,557 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 259 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,557 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,635 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 252 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:45,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,557 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:28,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,557 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,181 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,572 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | 2019-10-15 03:15:45,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 389 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:45,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,572 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 244 for changeToCandidate datanode_2 | 2019-10-15 03:15:28,635 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:45,208 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,663 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:135ms, electionTimeout:131ms datanode_1 | 2019-10-15 03:15:27,595 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection224: begin an election at term 245 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,214 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 390 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:28,663 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,214 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,605 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection224: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t245, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,663 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 266 for changeToCandidate datanode_3 | 2019-10-15 03:15:45,214 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,610 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection224: begin an election at term 246 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,663 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:45,214 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection224: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t246, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t246] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t246, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,664 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 253 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:45,289 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 391 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,637 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 246 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:28,664 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:45,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,289 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,317 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 402 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,755 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 260 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,664 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:45,317 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,755 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,664 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,317 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,755 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,317 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,755 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 403 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,759 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_2 | 2019-10-15 03:15:28,689 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection228: begin an election at term 267 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,716 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection228: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t267, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t267] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t267, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,759 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 246 for changeToCandidate datanode_2 | 2019-10-15 03:15:28,717 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 267 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:45,370 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:28,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:27,778 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection225: begin an election at term 247 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:45,502 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 404 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,806 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection225: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t247, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t247] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t247, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,819 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 254 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:45,502 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 392 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 247 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,503 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,857 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 261 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:45,503 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,819 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,503 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_1 | 2019-10-15 03:15:27,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,503 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,857 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,598 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 405 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 267 for changeToCandidate datanode_3 | 2019-10-15 03:15:45,598 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,889 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_2 | 2019-10-15 03:15:28,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:45,598 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection229: begin an election at term 268 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,598 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,889 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 247 for changeToCandidate datanode_2 | 2019-10-15 03:15:28,863 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 255 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:45,614 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 393 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:28,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:45,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:27,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:28,864 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,614 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection226: begin an election at term 248 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,889 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection229: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t268, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t268] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t268, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,864 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 394 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:27,920 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection226: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t248, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t248] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t248, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 268 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 248 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:28,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:45,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 406 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:27,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:27,994 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 262 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,891 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_3 | 2019-10-15 03:15:45,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:27,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,891 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 268 for changeToCandidate datanode_1 | 2019-10-15 03:15:27,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 407 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:27,995 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,905 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection230: begin an election at term 269 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:28,037 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_2 | 2019-10-15 03:15:28,920 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection230: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t269, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t269] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t269, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 395 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,920 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 269 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:28,037 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 248 for changeToCandidate datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:28,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,050 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection227: begin an election at term 249 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:45,937 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,962 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 256 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,068 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection227: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t249, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t249] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t249, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:28,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,068 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 249 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:28,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:45,937 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:28,962 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:45,995 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 396 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:45,995 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,226 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_3 | 2019-10-15 03:15:45,995 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:28,978 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 257 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,227 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:45,995 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:28,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,227 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 249 for changeToCandidate datanode_3 | 2019-10-15 03:15:46,081 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 408 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:28,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,227 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:46,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:28,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,241 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 263 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:46,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,023 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 258 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,241 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,082 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,242 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection228: begin an election at term 250 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,116 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 409 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:46,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,242 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,102 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_1 | 2019-10-15 03:15:28,267 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection228: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t250, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t250] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t250, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,116 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,267 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 250 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,169 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 410 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:29,102 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 269 for changeToCandidate datanode_3 | 2019-10-15 03:15:46,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,268 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:46,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,299 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 264 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:46,169 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,137 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection231: begin an election at term 270 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,186 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 397 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,156 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection231: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t270, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t270] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t270, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,187 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,300 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,187 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,314 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 265 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:46,187 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,256 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 411 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,314 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,156 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 270 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:28,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:46,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,156 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:28,315 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,256 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,156 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,406 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | 2019-10-15 03:15:46,314 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 398 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,231 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 259 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,314 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,406 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 250 for changeToCandidate datanode_3 | 2019-10-15 03:15:46,314 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:46,314 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,232 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,382 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 412 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection229: begin an election at term 251 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,261 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_3 | 2019-10-15 03:15:46,382 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,438 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection229: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t251, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t251] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t251, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,382 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,438 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 251 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,261 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 270 for changeToCandidate datanode_3 | 2019-10-15 03:15:46,382 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,438 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:29,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:46,456 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 399 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,438 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,497 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 266 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,457 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 413 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:46,457 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,497 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,283 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection232: begin an election at term 271 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,606 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | 2019-10-15 03:15:29,289 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection232: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t271, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t271] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t271, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,457 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 260 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,606 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 251 for changeToCandidate datanode_2 | 2019-10-15 03:15:29,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 271 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 400 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:29,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection230: begin an election at term 252 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 414 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:28,646 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection230: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t252, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,662 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection230: begin an election at term 253 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,690 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection230: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t253, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t253] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t253, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,318 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 253 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,678 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 271 for changeToCandidate datanode_3 | 2019-10-15 03:15:46,678 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 267 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 401 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,324 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection233: begin an election at term 272 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,690 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,339 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection233: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t272, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t272] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t272, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 415 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,790 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | 2019-10-15 03:15:29,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 272 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:46,886 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 253 for changeToCandidate datanode_2 | 2019-10-15 03:15:29,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:29,340 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 261 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:46,886 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,817 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection231: begin an election at term 254 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:46,949 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 402 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,833 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection231: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t254, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t254] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t254, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:46,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,340 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:46,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,834 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 254 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 262 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:46,949 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:29,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,047 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 416 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:47,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,847 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | 2019-10-15 03:15:29,465 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:47,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,493 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | 2019-10-15 03:15:47,047 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,847 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 254 for changeToCandidate datanode_2 | 2019-10-15 03:15:29,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,062 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 403 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:28,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:47,062 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,862 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection232: begin an election at term 255 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,493 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 272 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,062 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,863 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 268 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:47,062 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,509 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection234: begin an election at term 273 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:28,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,088 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 404 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,523 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection234: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t273, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:28,863 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:47,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,539 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection234: begin an election at term 274 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,562 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection234: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t274, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t274] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t274, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,088 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,889 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection232: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t255, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t255] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t255, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,562 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 274 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:47,157 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 405 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,562 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:47,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,889 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 255 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,562 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:47,157 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:29,646 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 263 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:47,157 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,646 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,244 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 417 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,906 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 269 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,646 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:47,244 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,673 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_3 | 2019-10-15 03:15:47,244 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,906 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,673 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,244 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,946 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_2 | 2019-10-15 03:15:29,673 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 274 for changeToCandidate datanode_1 | 2019-10-15 03:15:28,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,278 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 418 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,673 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:28,946 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 255 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,278 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,701 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection235: begin an election at term 275 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:28,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:47,278 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,715 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection235: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t275, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t275] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t275, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:28,960 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection233: begin an election at term 256 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,278 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,715 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 275 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:28,960 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection233: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t256, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,335 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 406 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:28,965 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection233: begin an election at term 257 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:47,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:28,991 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection233: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t257, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t257] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t257, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:29,799 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 264 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:47,335 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:28,991 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 257 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:29,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,420 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 419 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:28,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:29,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:47,420 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:28,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,799 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:47,420 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,015 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_2 | 2019-10-15 03:15:29,827 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | 2019-10-15 03:15:47,420 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,827 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,507 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 420 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,016 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 257 for changeToCandidate datanode_2 | 2019-10-15 03:15:29,827 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 275 for changeToCandidate datanode_1 | 2019-10-15 03:15:29,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:47,507 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection234: begin an election at term 258 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,827 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:29,038 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection234: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t258, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t258] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t258, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,508 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,831 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection236: begin an election at term 276 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,038 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 258 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:47,566 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 407 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,858 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection236: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t276, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t276] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t276, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:47,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,858 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 276 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:29,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:47,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:29,138 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 270 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:47,566 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,665 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 408 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,973 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_1 | 2019-10-15 03:15:29,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:29,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,665 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,138 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,973 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 276 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,665 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,206 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | 2019-10-15 03:15:29,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:47,665 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,986 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection237: begin an election at term 277 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,206 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,698 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 421 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:29,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 265 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,206 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 258 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,698 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,207 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:47,698 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,988 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:47,698 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,229 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection235: begin an election at term 259 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,757 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 409 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:29,993 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection237: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t277, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t277] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t277, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,258 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection235: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t259, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t259] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t259, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:29,993 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 277 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:29,258 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 259 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:47,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:29,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:29,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:47,757 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:29,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:47,779 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 422 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,051 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 266 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:47,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,279 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_2 | 2019-10-15 03:15:30,051 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,051 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,279 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 259 for changeToCandidate datanode_2 | 2019-10-15 03:15:30,051 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:47,779 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:30,062 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_3 | 2019-10-15 03:15:47,841 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 410 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,284 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 271 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,062 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 277 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:30,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:47,841 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,285 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,086 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection238: begin an election at term 278 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,872 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 411 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,288 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection236: begin an election at term 260 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,111 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection238: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t278, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t278] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t278, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:47,872 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,295 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection236: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t260, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t260] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t260, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,111 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 278 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:29,295 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 260 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:47,872 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:29,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:47,872 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:47,974 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 423 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 267 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,325 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | 2019-10-15 03:15:47,974 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:47,974 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,325 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 260 for changeToCandidate datanode_3 | 2019-10-15 03:15:47,974 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,112 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:48,069 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 412 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,224 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_1 | 2019-10-15 03:15:29,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 272 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:48,069 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,224 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,069 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,224 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 278 for changeToCandidate datanode_1 | 2019-10-15 03:15:29,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,069 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,224 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:29,326 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,138 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 413 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,237 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection239: begin an election at term 279 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,338 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection237: begin an election at term 261 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,138 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,255 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection239: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t279, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t279] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t279, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,138 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,255 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 279 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:30,255 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:48,138 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,255 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:48,175 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 414 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:48,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,259 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_1 | 2019-10-15 03:15:29,363 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection237: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t261, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t261] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t261, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,175 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,259 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 279 for changeToCandidate datanode_3 | 2019-10-15 03:15:48,190 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 424 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,364 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 261 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:30,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:48,190 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:30,290 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection240: begin an election at term 280 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,190 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:30,292 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 268 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,441 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_3 | 2019-10-15 03:15:48,190 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:48,222 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 425 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 261 for changeToCandidate datanode_2 | 2019-10-15 03:15:30,292 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,222 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,318 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection240: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t280, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t280] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t280, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,222 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 280 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:48,222 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection238: begin an election at term 262 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:48,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 426 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,507 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection238: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t262, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t262] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t262, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:48,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,507 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 262 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:30,318 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_1 | 2019-10-15 03:15:29,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:48,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,362 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 280 for changeToCandidate datanode_3 | 2019-10-15 03:15:48,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 415 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,511 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 273 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:48,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,331 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection241: begin an election at term 281 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,359 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection241: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t281, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t281] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t281, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,359 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 281 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:29,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 427 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 274 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,541 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 416 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,625 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,359 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 269 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,625 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 262 for changeToCandidate datanode_2 | 2019-10-15 03:15:30,360 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:30,360 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:48,499 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 417 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,645 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection239: begin an election at term 263 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,360 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,659 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection239: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t263, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t263] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t263, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_3 | 2019-10-15 03:15:48,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,660 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 263 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:30,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,499 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,470 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 281 for changeToCandidate datanode_1 | 2019-10-15 03:15:29,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:48,584 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 428 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:29,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,483 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection242: begin an election at term 282 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,702 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 275 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:48,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,505 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection242: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t282, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t282] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t282, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,584 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,505 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 282 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:29,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,636 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 418 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:29,702 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,769 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:15:30,537 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 270 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,770 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,770 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 263 for changeToCandidate datanode_3 | 2019-10-15 03:15:48,637 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,770 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:48,816 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 429 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,538 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,797 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection240: begin an election at term 264 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,670 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:163ms datanode_3 | 2019-10-15 03:15:48,816 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,830 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection240: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t264, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t264] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t264, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,817 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 282 for changeToCandidate datanode_1 | 2019-10-15 03:15:29,830 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 264 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:48,817 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:48,819 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 419 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,830 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:30,694 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection243: begin an election at term 283 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,830 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:30,722 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection243: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t283, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t283] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t283, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:48,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 276 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,723 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 283 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:48,819 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:29,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,904 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 430 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:29,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,774 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 271 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,973 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:15:48,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:48,938 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 420 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:29,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,938 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,973 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 264 for changeToCandidate datanode_2 | 2019-10-15 03:15:30,869 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_3 | 2019-10-15 03:15:48,938 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:30,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,938 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,986 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection241: begin an election at term 265 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,869 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 283 for changeToCandidate datanode_1 | 2019-10-15 03:15:29,987 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 277 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:48,982 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 431 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,883 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection244: begin an election at term 284 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:29,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:48,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,988 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:48,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:48,982 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:29,993 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection241: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t265, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t265] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t265, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:30,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection244: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t284, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t284] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t284, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,023 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 432 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:29,993 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 265 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:29,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:30,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 284 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:29,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,034 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_3 | 2019-10-15 03:15:49,030 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 421 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:30,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 265 for changeToCandidate datanode_3 | 2019-10-15 03:15:49,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,921 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_1 | 2019-10-15 03:15:30,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:49,030 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,049 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection242: begin an election at term 266 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,072 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 422 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,921 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 284 for changeToCandidate datanode_1 | 2019-10-15 03:15:30,065 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection242: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t266, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t266] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t266, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,072 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:30,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 266 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,072 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:30,935 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection245: begin an election at term 285 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:49,072 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,233 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 423 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:30,952 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection245: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t285, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t285] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t285, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 278 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:30,952 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 285 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,233 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,952 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:30,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:30,952 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,088 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:30,992 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 272 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:49,233 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,098 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_2 | 2019-10-15 03:15:30,992 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,233 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:30,992 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:49,235 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 433 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 266 for changeToCandidate datanode_2 | 2019-10-15 03:15:30,992 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:49,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:31,021 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_3 | 2019-10-15 03:15:49,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,110 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection243: begin an election at term 267 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:31,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,235 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,021 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 285 for changeToCandidate datanode_1 | 2019-10-15 03:15:30,138 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection243: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t267, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t267] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t267, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,387 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 424 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:49,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,047 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection246: begin an election at term 286 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,138 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 267 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,387 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 273 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:49,424 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 434 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,240 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 279 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:49,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,424 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,049 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,460 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 435 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,079 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection246: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t286, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t286] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t286, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,240 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:49,460 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 286 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:30,267 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | 2019-10-15 03:15:49,460 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:30,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,460 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,267 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 267 for changeToCandidate datanode_3 | 2019-10-15 03:15:49,537 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 425 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,119 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 274 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:49,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,290 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection244: begin an election at term 268 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:31,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,292 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 280 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:49,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,120 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,537 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,137 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 275 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:49,650 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 436 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,292 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:49,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,317 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection244: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t268, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t268] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t268, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:31,137 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:49,650 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,318 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 268 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:31,154 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | 2019-10-15 03:15:49,701 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 437 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,154 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,701 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 426 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,154 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 286 for changeToCandidate datanode_3 | 2019-10-15 03:15:49,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,154 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:49,701 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:31,167 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection247: begin an election at term 287 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,168 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection247: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t287, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,702 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,193 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection247: begin an election at term 288 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,333 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 281 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,208 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection247: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t288, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t288] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t288, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,702 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,702 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,333 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:49,748 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 438 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 288 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:30,339 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_3 | 2019-10-15 03:15:49,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:49,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,339 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 268 for changeToCandidate datanode_2 | 2019-10-15 03:15:31,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:49,748 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:49,858 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 427 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,358 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection245: begin an election at term 269 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,373 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection245: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t269, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t269] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t269, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:49,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,355 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 276 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,373 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 269 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:49,858 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:49,951 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 439 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:49,951 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 282 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:49,951 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,356 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,377 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_1 | 2019-10-15 03:15:30,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:49,951 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,485 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,377 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 288 for changeToCandidate datanode_3 | 2019-10-15 03:15:50,063 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 428 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,510 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_2 | 2019-10-15 03:15:31,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:50,063 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,063 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,383 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection248: begin an election at term 289 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:30,510 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 269 for changeToCandidate datanode_2 | 2019-10-15 03:15:31,402 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection248: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t289, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t289] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t289, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,064 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:31,402 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 289 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:50,079 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 429 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,402 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:50,079 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,402 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection246: begin an election at term 270 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,079 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,402 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 277 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:50,079 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,571 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection246: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t270, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t270] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t270, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:31,402 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 430 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,572 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 270 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:31,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:50,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:31,403 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:50,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,475 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 278 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:50,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,697 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 283 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,475 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,180 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 440 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,475 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:50,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,475 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,697 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:50,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,504 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 279 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,758 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_3 | 2019-10-15 03:15:50,180 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,504 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,334 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 431 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 270 for changeToCandidate datanode_2 | 2019-10-15 03:15:31,504 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:50,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:31,504 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:50,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,772 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection247: begin an election at term 271 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,334 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,790 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection247: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t271, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t271] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t271, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:31,535 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 280 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:50,361 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 441 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 271 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:31,535 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:50,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,536 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:50,361 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,568 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_1 | 2019-10-15 03:15:30,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:50,402 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 442 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:50,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,569 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 289 for changeToCandidate datanode_1 | 2019-10-15 03:15:30,885 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 284 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:50,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:30,885 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,402 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 443 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:30,885 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,574 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection249: begin an election at term 290 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 432 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,886 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,601 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection249: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t290, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t290] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t290, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,937 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 285 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,602 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 290 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,602 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:30,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,602 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,736 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 281 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,937 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,966 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_3 | 2019-10-15 03:15:50,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:30,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:30,966 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 271 for changeToCandidate datanode_3 | 2019-10-15 03:15:50,505 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 433 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:30,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:31,736 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:30,989 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection248: begin an election at term 272 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,505 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,766 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 282 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection248: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t272, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t272] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t272, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,506 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,006 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 272 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:50,506 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:50,690 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 434 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,766 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:50,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,789 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_1 | 2019-10-15 03:15:31,023 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | 2019-10-15 03:15:31,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,690 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 444 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 290 for changeToCandidate datanode_1 | 2019-10-15 03:15:31,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 272 for changeToCandidate datanode_3 | 2019-10-15 03:15:50,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:50,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,792 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection250: begin an election at term 291 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,691 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,822 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection250: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t291, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t291] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t291, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,691 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:31,822 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 291 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:50,691 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:31,047 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection249: begin an election at term 273 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,855 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 435 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:50,856 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 286 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,873 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 283 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:50,856 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,873 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,856 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,873 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:50,916 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 445 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,049 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,873 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_3 | 2019-10-15 03:15:50,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,873 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,079 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection249: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t273, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t273] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t273, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:50,916 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,873 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,079 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 273 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:50,916 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,874 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 291 for changeToCandidate datanode_1 | 2019-10-15 03:15:31,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:50,982 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 436 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:31,874 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:31,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:50,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection251: begin an election at term 292 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,095 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_3 | 2019-10-15 03:15:50,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:31,922 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection251: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t292, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t292] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t292, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,095 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:50,982 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,923 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 292 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:31,095 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 273 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,145 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 446 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,923 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:31,095 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:51,145 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,923 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,118 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection250: begin an election at term 274 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,145 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,131 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection250: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t274, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,145 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,927 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_1 | 2019-10-15 03:15:31,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection250: begin an election at term 275 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,176 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 447 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,152 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection250: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t275, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t275] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t275, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,176 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,152 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 275 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:31,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,176 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,152 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,176 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,152 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 292 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,210 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 437 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,169 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 287 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:51,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,951 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection252: begin an election at term 293 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:31,982 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection252: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t293, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t293] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t293, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,169 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:31,982 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 293 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,210 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,194 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 288 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:31,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,331 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 438 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:31,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:51,331 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:51,331 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,060 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 284 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,195 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:51,331 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,060 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,332 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 448 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,328 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_2 | 2019-10-15 03:15:32,061 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:51,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,328 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,061 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,329 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 275 for changeToCandidate datanode_2 | 2019-10-15 03:15:32,149 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_1 | 2019-10-15 03:15:31,329 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:31,353 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection251: begin an election at term 276 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:32,149 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,382 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection251: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t276, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t276] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t276, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,149 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 293 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,333 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:51,348 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 449 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,382 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 276 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:32,154 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection253: begin an election at term 294 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:32,158 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection253: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t294, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,169 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection253: begin an election at term 295 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,348 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 289 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,197 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection253: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t295, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t295] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t295, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,454 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 450 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,198 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 295 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,454 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,198 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,454 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,385 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,198 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,390 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:15:32,207 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 285 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:51,454 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,391 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,391 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 276 for changeToCandidate datanode_2 | 2019-10-15 03:15:32,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:51,528 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 439 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,207 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,391 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:51,528 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,273 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | 2019-10-15 03:15:51,528 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,401 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection252: begin an election at term 277 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:32,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,528 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,417 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection252: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t277, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t277] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t277, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:32,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 295 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,648 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 451 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:51,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,281 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection254: begin an election at term 296 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:51,648 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,301 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection254: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t296, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t296] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t296, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,417 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 277 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,721 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 440 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,417 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:32,301 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 296 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,721 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,417 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:51,721 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,460 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | 2019-10-15 03:15:32,301 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,721 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,301 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,723 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 452 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,352 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_1 | 2019-10-15 03:15:31,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 277 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:32,352 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 296 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,723 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,473 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection253: begin an election at term 278 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,798 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 441 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,480 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection253: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t278, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t278] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t278, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:32,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:51,798 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,480 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 278 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:32,369 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection255: begin an election at term 297 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,798 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:32,406 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection255: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t297, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t297] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t297, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,406 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 297 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,798 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,485 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_2 | 2019-10-15 03:15:32,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,830 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 442 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:51,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 278 for changeToCandidate datanode_3 | 2019-10-15 03:15:51,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:51,830 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,411 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_1 | 2019-10-15 03:15:31,502 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection254: begin an election at term 279 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 453 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,511 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection254: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t279, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t279] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t279, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:51,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,411 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 297 for changeToCandidate datanode_1 | 2019-10-15 03:15:31,511 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 279 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:51,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:31,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:51,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,431 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection256: begin an election at term 298 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:51,983 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 443 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection256: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t298, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t298] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t298, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,529 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_3 | 2019-10-15 03:15:51,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 298 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:31,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:31,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 279 for changeToCandidate datanode_2 | 2019-10-15 03:15:32,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:51,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:15:51,983 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 286 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:52,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 444 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:52,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,534 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection255: begin an election at term 280 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,479 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,549 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection255: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t280, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t280] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t280, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,013 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,479 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_1 | 2019-10-15 03:15:31,549 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 280 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,149 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 445 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,549 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:52,149 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 454 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 298 for changeToCandidate datanode_1 | 2019-10-15 03:15:31,549 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,576 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 290 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:52,149 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:52,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,576 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,493 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection257: begin an election at term 299 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,576 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,516 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection257: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t299, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t299] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t299, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,516 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 299 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:31,576 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:52,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,709 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_3 | 2019-10-15 03:15:52,358 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 455 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,709 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,516 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,709 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 280 for changeToCandidate datanode_2 | 2019-10-15 03:15:32,517 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,577 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_1 | 2019-10-15 03:15:31,709 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:32,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,734 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection256: begin an election at term 281 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,358 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,577 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 299 for changeToCandidate datanode_1 | 2019-10-15 03:15:31,745 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection256: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t281, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,764 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection256: begin an election at term 282 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,374 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 446 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:31,790 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection256: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t282, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t282] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t282, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 282 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,583 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection258: begin an election at term 300 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:52,374 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,609 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection258: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t300, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t300] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t300, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,483 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 456 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,609 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 300 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,609 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:52,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,609 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,794 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 291 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:52,483 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,697 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 287 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:52,515 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 447 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,515 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,794 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,697 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,845 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_3 | 2019-10-15 03:15:52,515 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 288 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,845 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:52,515 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,846 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 282 for changeToCandidate datanode_2 | 2019-10-15 03:15:32,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:31,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:32,722 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,871 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection257: begin an election at term 283 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,554 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 448 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:15:31,900 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection257: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t283, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t283] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t283, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,554 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,900 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 283 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,554 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:32,774 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 300 for changeToCandidate datanode_3 | 2019-10-15 03:15:52,554 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:52,669 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 457 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,792 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection259: begin an election at term 301 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,669 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,807 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection259: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t301, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t301] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t301, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:31,901 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 292 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,807 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 301 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,669 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:52,670 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:52,783 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 449 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:31,902 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,812 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | 2019-10-15 03:15:52,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:31,955 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 293 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,812 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:52,783 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:31,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,812 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 301 for changeToCandidate datanode_3 | 2019-10-15 03:15:52,783 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:31,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:32,812 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:52,837 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 458 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:31,955 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,892 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection260: begin an election at term 302 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,838 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,048 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_2 | 2019-10-15 03:15:32,909 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection260: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t302, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t302] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t302, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,838 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,909 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 302 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:52,838 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:32,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:32,048 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 283 for changeToCandidate datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 450 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:32,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:32,936 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 289 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,058 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection258: begin an election at term 284 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:32,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,078 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection258: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t284, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t284] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t284, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 459 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:32,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,078 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 284 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:32,936 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:33,029 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 290 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 294 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:52,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,157 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:52,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,157 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:52,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,157 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:52,907 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 460 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:32,171 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 295 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:52,907 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,171 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,029 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:52,907 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,171 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,082 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 291 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,171 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:52,907 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,082 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,201 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | 2019-10-15 03:15:53,015 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 451 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,082 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,015 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,082 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 284 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,016 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,089 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_1 | 2019-10-15 03:15:32,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:53,016 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,205 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection259: begin an election at term 285 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 452 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,089 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 302 for changeToCandidate datanode_1 | 2019-10-15 03:15:32,228 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection259: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t285, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t285] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t285, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:32,228 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 285 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,095 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection261: begin an election at term 303 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:32,228 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:53,075 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,097 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 292 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,228 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 461 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,283 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 296 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,097 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,098 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,075 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,284 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,112 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection261: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t303, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t303] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t303, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,237 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 462 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:32,371 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 297 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 303 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,237 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:53,237 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,237 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 293 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:53,267 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 453 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,372 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:53,267 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,426 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_3 | 2019-10-15 03:15:53,267 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,267 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,152 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_1 | 2019-10-15 03:15:32,426 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 285 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,311 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 463 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:53,311 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,152 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 303 for changeToCandidate datanode_1 | 2019-10-15 03:15:32,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 298 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:53,311 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,152 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:53,312 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,156 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection262: begin an election at term 304 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 464 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:32,433 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,165 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection262: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t304, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t304] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t304, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection260: begin an election at term 286 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,165 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 304 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:32,494 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection260: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t286, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t286] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t286, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,367 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,494 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 286 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,462 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 454 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,180 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 294 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:53,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,462 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,495 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 299 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,180 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:53,537 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 455 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,193 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_1 | 2019-10-15 03:15:32,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:53,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,495 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 304 for changeToCandidate datanode_1 | 2019-10-15 03:15:32,585 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 300 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:32,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:53,537 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,207 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection263: begin an election at term 305 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,553 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 465 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:32,585 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,223 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection263: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t305, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t305] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t305, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,672 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:15:33,223 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 305 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:32,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,553 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:32,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 286 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,555 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 456 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 295 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:53,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection261: begin an election at term 287 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,555 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,717 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection261: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t287, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,298 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:53,697 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 457 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,719 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection261: begin an election at term 288 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,356 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_1 | 2019-10-15 03:15:32,742 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection261: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t288, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t288] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t288, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,743 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 288 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:33,356 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 305 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,743 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:53,712 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 458 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:32,794 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 301 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,371 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection264: begin an election at term 306 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,712 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,388 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection264: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t306, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t306] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t306, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:32,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:53,713 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,388 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 306 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,713 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:32,794 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:53,750 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 466 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,894 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 302 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,443 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 296 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:53,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,751 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,895 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 467 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:32,895 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,443 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:32,921 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 459 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,500 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_1 | 2019-10-15 03:15:32,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,921 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 288 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,500 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:32,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,500 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 306 for changeToCandidate datanode_3 | 2019-10-15 03:15:53,898 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,500 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:32,934 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection262: begin an election at term 289 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,514 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection265: begin an election at term 307 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:32,950 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection262: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t289, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t289] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t289, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:53,898 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,542 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection265: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t307, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t307] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t307, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:32,951 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 289 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:53,954 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 468 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,542 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 307 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:32,951 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:53,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:53,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:32,951 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:53,954 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,017 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:33,655 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_3 | 2019-10-15 03:15:54,020 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 460 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,018 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,020 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,655 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 307 for changeToCandidate datanode_1 | 2019-10-15 03:15:33,018 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 289 for changeToCandidate datanode_3 | 2019-10-15 03:15:54,020 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:33,018 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:54,020 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,657 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 297 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,027 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection263: begin an election at term 290 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,091 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 469 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,043 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection263: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t290, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t290] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t290, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,091 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,043 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 290 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,657 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,043 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,092 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,044 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,671 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection266: begin an election at term 308 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,135 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 461 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,053 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_3 | 2019-10-15 03:15:54,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection266: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t308, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t308] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t308, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,688 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 308 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:33,053 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 290 for changeToCandidate datanode_3 | 2019-10-15 03:15:54,136 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:33,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:54,228 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 470 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:33,079 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection264: begin an election at term 291 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:54,228 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,089 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection264: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t291, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,718 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | 2019-10-15 03:15:54,228 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,718 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,095 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection264: begin an election at term 292 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,228 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,718 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 308 for changeToCandidate datanode_1 | 2019-10-15 03:15:33,097 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 303 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:54,276 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 471 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,718 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:54,276 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,732 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection267: begin an election at term 309 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,276 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection267: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t309, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t309] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t309, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,097 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,276 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,749 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 309 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,279 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 462 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:33,112 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection264: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t292, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t292] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t292, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,112 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 292 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:33,866 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 298 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,279 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:54,307 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 463 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,116 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | 2019-10-15 03:15:54,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,116 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,866 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,116 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 292 for changeToCandidate datanode_2 | 2019-10-15 03:15:33,938 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:54,307 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:33,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:54,322 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 464 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:33,938 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 309 for changeToCandidate datanode_1 | 2019-10-15 03:15:33,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection265: begin an election at term 293 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:54,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,127 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection265: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t293, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t293] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t293, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:33,951 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection268: begin an election at term 310 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,322 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,127 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 293 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:33,967 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection268: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t310, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t310] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t310, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,432 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 472 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:33,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:33,968 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 310 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,158 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 304 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:33,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,432 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:33,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:54,485 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 465 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,017 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 299 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:54,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,159 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,174 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_3 | 2019-10-15 03:15:54,485 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,018 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,018 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,485 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,175 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 293 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,019 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_1 | 2019-10-15 03:15:33,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,558 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 473 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:33,178 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection266: begin an election at term 294 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,019 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 310 for changeToCandidate datanode_3 | 2019-10-15 03:15:54,558 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,185 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection266: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t294, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t294] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t294, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:54,558 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 294 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,558 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:34,031 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection269: begin an election at term 311 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,638 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 466 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,047 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection269: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t311, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t311] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t311, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,638 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,209 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 305 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,047 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 311 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:33,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,638 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,047 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:33,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:54,638 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,047 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,209 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,788 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 474 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_1 | 2019-10-15 03:15:33,282 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_3 | 2019-10-15 03:15:54,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,153 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 311 for changeToCandidate datanode_3 | 2019-10-15 03:15:54,788 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,282 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 294 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 467 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,283 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,167 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection270: begin an election at term 312 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection267: begin an election at term 295 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,169 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 300 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 475 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,169 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection267: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t295, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t295] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t295, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,169 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,324 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 295 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,170 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,184 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection270: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t312, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t312] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t312, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,849 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,184 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 312 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:33,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,849 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:54,898 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 468 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:54,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,185 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,373 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 306 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:54,898 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,253 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 301 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,898 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 469 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:34,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,373 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,253 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,427 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_2 | 2019-10-15 03:15:34,255 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 476 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:33,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,255 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,427 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 295 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,255 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 312 for changeToCandidate datanode_3 | 2019-10-15 03:15:54,919 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,255 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:54,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection268: begin an election at term 296 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,278 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection271: begin an election at term 313 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:54,919 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,467 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection268: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t296, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t296] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t296, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,295 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection271: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t313, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t313] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t313, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,123 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 470 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,467 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 296 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:34,295 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 313 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:33,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:33,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,124 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 307 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:55,124 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,459 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 302 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:55,142 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 471 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,459 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,516 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,459 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:55,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,641 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_3 | 2019-10-15 03:15:55,142 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,641 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,459 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:55,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 477 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:33,641 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 296 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,482 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_3 | 2019-10-15 03:15:55,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,641 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection269: begin an election at term 297 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,672 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection269: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t297, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t297] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t297, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,482 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 313 for changeToCandidate datanode_3 | 2019-10-15 03:15:55,219 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 472 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 297 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:34,483 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:55,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,496 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection272: begin an election at term 314 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,219 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,514 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection272: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t314, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t314] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t314, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,673 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 308 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,514 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 314 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:33,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,289 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 478 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:33,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,673 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:55,289 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,696 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 303 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:55,289 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,734 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 309 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,696 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,358 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 473 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,696 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:33,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,696 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,734 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,698 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_1 | 2019-10-15 03:15:33,843 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_2 | 2019-10-15 03:15:34,698 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,358 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,698 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 314 for changeToCandidate datanode_1 | 2019-10-15 03:15:33,843 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 297 for changeToCandidate datanode_3 | 2019-10-15 03:15:55,358 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,698 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:33,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,703 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection273: begin an election at term 315 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,864 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection270: begin an election at term 298 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,358 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,711 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection273: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t315, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:33,881 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection270: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t298, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t298] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t298, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,483 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 479 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,729 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection273: begin an election at term 316 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,881 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 298 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:34,745 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection273: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t316, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t316] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t316, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:33,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:34,745 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 316 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:55,483 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:33,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,745 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,522 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 474 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:33,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 310 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,745 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:55,522 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,784 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_1 | 2019-10-15 03:15:33,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,522 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:33,953 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:55,523 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,784 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 316 for changeToCandidate datanode_1 | 2019-10-15 03:15:34,002 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_3 | 2019-10-15 03:15:55,575 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 475 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:34,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:55,575 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,797 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection274: begin an election at term 317 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,575 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,002 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 298 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,825 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection274: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t317, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t317] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t317, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,575 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:34,825 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 317 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:55,688 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 480 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection271: begin an election at term 299 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,825 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection271: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t299, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t299] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t299, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:34,825 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:55,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 299 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:34,839 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 304 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,782 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 476 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:34,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 311 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:55,782 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,839 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:55,782 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,914 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_3 | 2019-10-15 03:15:55,782 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,033 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,914 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,894 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 481 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,161 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:15:34,914 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 317 for changeToCandidate datanode_3 | 2019-10-15 03:15:55,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,161 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,914 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:55,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,161 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 299 for changeToCandidate datanode_2 | 2019-10-15 03:15:34,927 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection275: begin an election at term 318 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:55,895 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,947 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection275: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t318, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t318] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t318, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,167 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection272: begin an election at term 300 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,925 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 477 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:34,947 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 318 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:34,170 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 312 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:55,925 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:34,170 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,170 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:34,948 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 305 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,170 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,948 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,184 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection272: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t300, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t300] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t300, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:55,925 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 300 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:55,925 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,948 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:55,981 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 482 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,948 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:55,981 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,237 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | 2019-10-15 03:15:34,975 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_3 | 2019-10-15 03:15:55,981 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:34,976 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:55,981 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:34,976 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 318 for changeToCandidate datanode_1 | 2019-10-15 03:15:34,237 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 300 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,080 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 483 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:34,976 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:34,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:56,080 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:34,986 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection276: begin an election at term 319 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,251 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection273: begin an election at term 301 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,080 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection276: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t319, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t319] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t319, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection273: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t301, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t301] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t301, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,080 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,005 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 319 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,121 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 478 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:35,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,121 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 301 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,163 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 306 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,163 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,163 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:56,168 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 484 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 313 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,163 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:56,168 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:56,168 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,174 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_1 | 2019-10-15 03:15:34,280 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,168 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,408 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:15:35,175 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 319 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 479 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,409 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 301 for changeToCandidate datanode_2 | 2019-10-15 03:15:35,185 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection277: begin an election at term 320 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 485 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,204 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection277: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t320, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t320] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t320, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,204 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 320 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:34,457 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection274: begin an election at term 302 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:34,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection274: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t302, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t302] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t302, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,270 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 302 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,271 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,224 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_1 | 2019-10-15 03:15:34,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,271 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,224 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,325 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 486 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,498 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 314 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,224 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 320 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,224 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:34,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,231 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection278: begin an election at term 321 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:56,325 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection278: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t321, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t321] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t321, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,498 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:56,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 487 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,242 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 321 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:34,680 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_3 | 2019-10-15 03:15:56,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,242 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:34,680 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,242 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,680 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 302 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,680 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:56,441 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 488 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,243 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 307 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,694 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection275: begin an election at term 303 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,704 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection275: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t303, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t303] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t303, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,441 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,704 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 303 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,441 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 480 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:35,243 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:56,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:15:56,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,705 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 315 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:56,470 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,705 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,584 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 489 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,705 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,350 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 321 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,705 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:56,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 316 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,365 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection279: begin an election at term 322 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,585 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,393 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection279: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t322, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t322] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t322, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,393 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 322 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,586 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 481 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,731 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,799 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 317 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:56,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,799 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,471 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 308 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:56,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,799 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,765 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 482 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,799 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:56,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,471 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | 2019-10-15 03:15:56,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,582 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:15:56,765 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,766 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 490 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 303 for changeToCandidate datanode_2 | 2019-10-15 03:15:35,582 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 322 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:35,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:56,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,837 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection276: begin an election at term 304 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:35,637 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection280: begin an election at term 323 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:34,845 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection276: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t304, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t304] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t304, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:35,639 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 309 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:56,828 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 491 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:34,845 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 304 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,828 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,845 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,829 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,845 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:56,829 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,640 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,925 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_3 | 2019-10-15 03:15:56,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 492 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,664 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection280: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t323, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t323] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t323, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,925 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:56,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,664 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 323 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:34,925 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 304 for changeToCandidate datanode_3 | 2019-10-15 03:15:56,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,664 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:34,925 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:56,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:34,928 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 318 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,664 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,928 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,665 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_3 | 2019-10-15 03:15:56,957 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 483 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,928 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:35,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,929 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:56,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:34,946 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection277: begin an election at term 305 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,665 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 323 for changeToCandidate datanode_1 | 2019-10-15 03:15:34,987 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection277: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t305, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t305] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t305, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:56,957 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:34,987 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 305 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:56,999 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 493 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,670 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection281: begin an election at term 324 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:56,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,691 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection281: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t324, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t324] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t324, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:34,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:56,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,691 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 324 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:34,988 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 319 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:56,999 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:34,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 484 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:35,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:34,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,827 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 310 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:34,988 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,827 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,153 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_3 | 2019-10-15 03:15:57,078 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,153 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,828 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 494 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,153 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 305 for changeToCandidate datanode_2 | 2019-10-15 03:15:35,841 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 311 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,153 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:35,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,161 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection278: begin an election at term 306 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:35,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:57,078 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,183 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection278: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t306, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t306] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t306, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,106 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 495 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,841 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,183 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 306 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:57,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,866 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_1 | 2019-10-15 03:15:35,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:57,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,183 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,106 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,187 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 320 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,866 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 324 for changeToCandidate datanode_3 | 2019-10-15 03:15:57,266 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 485 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,187 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,866 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:57,267 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,188 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,267 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,876 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection282: begin an election at term 325 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,267 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,886 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection282: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t325, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t325] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t325, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:35,188 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,332 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 486 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:35,887 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 325 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:35,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 321 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:57,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,332 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,332 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,912 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 312 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,233 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_3 | 2019-10-15 03:15:57,344 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 496 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:35,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,344 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:57,344 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,233 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,913 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,344 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 306 for changeToCandidate datanode_2 | 2019-10-15 03:15:35,955 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_3 | 2019-10-15 03:15:57,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 497 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:35,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,241 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection279: begin an election at term 307 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:35,955 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 325 for changeToCandidate datanode_3 | 2019-10-15 03:15:57,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection279: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t307, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t307] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t307, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:35,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:57,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 307 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:35,970 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection283: begin an election at term 326 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:35,987 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection283: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t326, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t326] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t326, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,456 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 498 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,456 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,367 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 322 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:57,456 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:35,987 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 326 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:35,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,456 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:35,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,563 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 487 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:35,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,367 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:35,999 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_3 | 2019-10-15 03:15:57,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,454 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | 2019-10-15 03:15:35,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:35,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 326 for changeToCandidate datanode_3 | 2019-10-15 03:15:57,563 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,454 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 307 for changeToCandidate datanode_2 | 2019-10-15 03:15:36,000 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:57,623 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 499 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:36,013 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection284: begin an election at term 327 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,623 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,469 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection280: begin an election at term 308 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,028 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection284: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t327, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t327] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t327, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,623 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection280: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t308, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t308] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t308, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,029 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 327 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:57,623 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 308 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:57,720 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 488 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:36,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,720 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,097 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 313 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,589 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_3 | 2019-10-15 03:15:57,720 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,097 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,720 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,589 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 308 for changeToCandidate datanode_2 | 2019-10-15 03:15:36,097 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:57,722 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 500 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:36,097 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection281: begin an election at term 309 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,176 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_3 | 2019-10-15 03:15:57,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,639 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 323 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:57,722 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,176 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,639 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,765 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 501 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,176 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 327 for changeToCandidate datanode_1 | 2019-10-15 03:15:35,640 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:57,765 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,176 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:35,640 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,192 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection285: begin an election at term 328 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:35,664 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection281: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t309, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t309] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t309, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,208 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection285: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t328, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t328] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t328, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:57,816 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 489 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,664 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 309 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 328 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:57,816 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:36,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:57,816 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:57,816 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 324 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,338 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | 2019-10-15 03:15:57,841 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 502 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,842 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 328 for changeToCandidate datanode_3 | 2019-10-15 03:15:57,842 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,673 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:57,842 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,783 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_2 | 2019-10-15 03:15:36,344 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 314 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:57,918 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 503 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,344 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:57,918 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,783 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 309 for changeToCandidate datanode_2 | 2019-10-15 03:15:36,344 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:57,918 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:36,344 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:57,918 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,825 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection282: begin an election at term 310 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,345 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection286: begin an election at term 329 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,037 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 490 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,834 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection282: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t310, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,362 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection286: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t329, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t329] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t329, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,037 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,839 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection282: begin an election at term 311 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,037 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,362 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 329 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:35,849 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection282: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t311, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t311] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t311, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,037 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:35,849 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 311 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:58,066 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 504 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:58,066 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,066 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,384 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 315 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:58,066 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,877 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 325 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:58,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 505 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,384 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,877 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,384 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:58,120 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,902 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | 2019-10-15 03:15:36,384 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:58,262 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 491 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:36,480 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_1 | 2019-10-15 03:15:35,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,262 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,480 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,902 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 311 for changeToCandidate datanode_3 | 2019-10-15 03:15:58,262 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,480 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 329 for changeToCandidate datanode_3 | 2019-10-15 03:15:58,262 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:36,480 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:58,319 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 506 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:35,911 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection283: begin an election at term 312 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,524 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection287: begin an election at term 330 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,319 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,929 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection283: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t312, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t312] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t312, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,549 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection287: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t330, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t330] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t330, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,319 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:35,929 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 312 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 330 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:58,319 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:35,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:36,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:58,386 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 492 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,929 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:58,386 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:35,972 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 326 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:58,386 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,585 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_1 | 2019-10-15 03:15:35,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,386 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,585 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:35,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:58,448 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 493 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:35,972 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,586 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 330 for changeToCandidate datanode_3 | 2019-10-15 03:15:58,448 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,014 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 327 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:58,448 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,609 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection288: begin an election at term 331 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 316 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:58,448 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,014 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,499 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 507 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:58,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,081 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | 2019-10-15 03:15:36,611 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:58,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,637 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection288: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t331, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t331] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t331, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,499 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,081 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 312 for changeToCandidate datanode_2 | 2019-10-15 03:15:36,637 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 331 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:36,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:58,618 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 494 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:36,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:58,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,095 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection284: begin an election at term 313 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:58,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection284: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t313, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t313] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t313, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:36,655 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 317 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:58,619 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,120 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 313 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,642 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 508 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:58,642 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,655 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:58,642 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,788 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_3 | 2019-10-15 03:15:58,642 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,194 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 328 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,714 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 495 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:36,788 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 331 for changeToCandidate datanode_3 | 2019-10-15 03:15:58,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:36,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:58,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,799 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection289: begin an election at term 332 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,194 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:58,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,304 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_2 | 2019-10-15 03:15:36,801 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 318 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:36,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,738 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 509 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,304 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 313 for changeToCandidate datanode_3 | 2019-10-15 03:15:58,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:58,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:36,801 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,342 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection285: begin an election at term 314 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,738 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,347 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 329 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:36,806 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection289: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t332, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,773 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 496 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:36,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,818 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection289: begin an election at term 333 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,773 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,826 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection289: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t333, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t333] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t333, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,773 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,347 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:58,773 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 333 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:36,362 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection285: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t314, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t314] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t314, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:58,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 497 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:36,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:58,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,362 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 314 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:58,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:36,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 319 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:58,903 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:36,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:58,923 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 510 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:36,362 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_2 | 2019-10-15 03:15:36,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:58,923 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,964 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:58,923 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,363 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 314 for changeToCandidate datanode_2 | 2019-10-15 03:15:36,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | 2019-10-15 03:15:58,923 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:59,056 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 498 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:36,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,382 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection286: begin an election at term 315 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,056 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:36,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 333 for changeToCandidate datanode_1 | 2019-10-15 03:15:36,390 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection286: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t315, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t315] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t315, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,056 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,390 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 315 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:36,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:59,056 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:36,994 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection290: begin an election at term 334 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,098 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 511 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:36,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:37,011 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection290: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t334, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t334] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t334, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,098 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,012 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 334 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,098 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,526 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 330 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:59,099 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:59,283 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 499 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:37,072 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_1 | 2019-10-15 03:15:36,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,526 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:59,284 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,072 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 334 for changeToCandidate datanode_1 | 2019-10-15 03:15:36,576 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_3 | 2019-10-15 03:15:59,284 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:36,576 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 500 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:37,081 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection291: begin an election at term 335 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,576 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 315 for changeToCandidate datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,576 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 512 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,098 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection291: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t335, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t335] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t335, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection287: begin an election at term 316 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,098 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 335 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,611 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 331 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,098 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,098 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,611 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,099 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 320 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,099 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection287: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t316, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t316] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t316, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,099 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,637 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 316 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,099 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:59,352 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 513 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,215 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_1 | 2019-10-15 03:15:36,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,352 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,215 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,639 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | 2019-10-15 03:15:37,215 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 335 for changeToCandidate datanode_3 | 2019-10-15 03:15:59,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,639 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,215 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:59,353 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,639 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 316 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,262 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection292: begin an election at term 336 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,367 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 501 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:36,639 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:59,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,653 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection288: begin an election at term 317 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,275 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection292: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t336, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t336] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t336, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection288: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t317, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t317] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t317, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,275 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 336 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,367 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 317 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:37,275 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:59,367 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:37,275 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:59,451 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 514 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:36,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,276 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 321 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:59,451 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,276 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,785 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_3 | 2019-10-15 03:15:59,451 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,276 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,451 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,785 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 317 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,276 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:15:59,486 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 515 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,353 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 322 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:36,799 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection289: begin an election at term 318 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,486 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,801 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 332 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:59,486 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,486 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,801 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,532 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 516 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,405 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_1 | 2019-10-15 03:15:36,801 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:59,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection289: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t318, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t318] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t318, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 318 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:37,405 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 336 for changeToCandidate datanode_3 | 2019-10-15 03:15:59,532 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:37,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:59,539 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 517 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:59,539 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,420 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection293: begin an election at term 337 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,539 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,424 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection293: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t337, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,539 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,437 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection293: begin an election at term 338 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,548 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 518 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:36,820 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 333 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:59,548 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,820 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,548 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 323 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:15:59,548 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,820 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,570 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 502 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:36,820 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:59,570 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,920 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_2 | 2019-10-15 03:15:37,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:15:59,570 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection293: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t338, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t338] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t338, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,570 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 318 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,454 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 338 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,613 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 519 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:36,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:37,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:59,613 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,962 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection290: begin an election at term 319 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:59,613 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,980 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection290: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t319, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t319] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t319, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,503 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_3 | 2019-10-15 03:15:59,613 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:36,980 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 319 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:37,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,657 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 503 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:36,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:37,503 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 338 for changeToCandidate datanode_3 | 2019-10-15 03:15:59,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:59,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:36,996 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 334 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:59,657 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,518 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection294: begin an election at term 339 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:15:59,688 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 504 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:37,535 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection294: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t339, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t339] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t339, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:36,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:15:59,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:36,996 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,536 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 339 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:15:59,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,083 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 335 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:15:59,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,083 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:15:59,796 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 520 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,083 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:15:59,796 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,084 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,573 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_2 | 2019-10-15 03:15:37,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,088 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_3 | 2019-10-15 03:15:59,796 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,573 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 339 for changeToCandidate datanode_3 | 2019-10-15 03:15:59,796 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:15:59,921 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 505 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 319 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,588 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection295: begin an election at term 340 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,089 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:37,617 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection295: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t340, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t340] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t340, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,921 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,096 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection291: begin an election at term 320 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,922 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,617 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 340 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:37,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection291: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t320, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t320] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t320, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:15:59,922 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,617 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:37,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 320 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:00,013 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 521 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:00,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,013 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,256 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:135ms datanode_3 | 2019-10-15 03:16:00,013 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,618 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,051 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 522 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 320 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 324 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:00,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:00,051 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,265 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 336 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:00,051 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,106 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 506 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,265 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,732 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 325 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:00,107 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,265 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,732 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,107 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,265 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,732 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,107 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,273 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection292: begin an election at term 321 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,732 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,218 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 507 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:00,218 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 326 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,285 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection292: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t321, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t321] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t321, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,218 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,285 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 321 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:00,219 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,285 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:00,220 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 523 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,765 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,285 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,816 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_1 | 2019-10-15 03:15:37,331 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_3 | 2019-10-15 03:16:00,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,816 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,220 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,332 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 321 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,816 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 340 for changeToCandidate datanode_3 | 2019-10-15 03:16:00,360 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 524 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:37,816 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:00,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,351 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection293: begin an election at term 322 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:37,829 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection296: begin an election at term 341 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,360 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,367 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection293: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t322, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t322] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t322, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,360 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,837 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection296: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t341, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t341] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t341, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,367 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 322 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:00,376 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 508 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:37,837 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 341 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:37,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:00,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,837 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:37,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 337 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,837 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,376 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 327 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,531 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 525 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,531 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,531 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,429 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_2 | 2019-10-15 03:15:37,882 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_3 | 2019-10-15 03:16:00,531 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,583 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 509 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,429 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 322 for changeToCandidate datanode_2 | 2019-10-15 03:15:37,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 341 for changeToCandidate datanode_3 | 2019-10-15 03:16:00,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,883 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:00,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:00,583 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,897 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection297: begin an election at term 342 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,437 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection294: begin an election at term 323 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,647 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 510 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,439 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 338 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:00,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,915 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection297: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t342, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t342] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t342, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,915 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 342 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:37,439 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,647 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,915 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:37,439 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,722 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 526 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,915 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,454 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection294: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t323, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t323] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t323, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,938 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 328 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,454 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 323 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:00,722 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:37,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:37,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,722 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,938 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,813 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 511 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,520 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 339 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:37,965 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 329 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:00,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:37,965 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:37,965 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,813 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:37,965 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,520 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,849 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 512 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,011 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_1 | 2019-10-15 03:15:37,590 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 340 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:00,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,590 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,012 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 342 for changeToCandidate datanode_1 | 2019-10-15 03:15:37,590 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:00,850 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:37,590 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:00,952 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 527 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,025 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection298: begin an election at term 343 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,653 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_3 | 2019-10-15 03:16:00,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,039 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection298: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t343, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t343] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t343, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,653 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,039 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 343 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:00,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:00,953 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,653 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 323 for changeToCandidate datanode_2 | 2019-10-15 03:15:38,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:00,986 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 513 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,653 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,055 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_3 | 2019-10-15 03:16:00,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection295: begin an election at term 324 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,055 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:00,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,678 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection295: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t324, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t324] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t324, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:00,986 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,055 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 343 for changeToCandidate datanode_1 | 2019-10-15 03:15:37,678 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 324 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,129 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 528 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:37,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:38,062 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection299: begin an election at term 344 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,129 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,072 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection299: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t344, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t344] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t344, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,129 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,717 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_2 | 2019-10-15 03:15:38,072 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 344 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,129 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,228 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 514 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,717 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 324 for changeToCandidate datanode_2 | 2019-10-15 03:15:38,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:01,228 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,079 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_3 | 2019-10-15 03:16:01,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,730 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection296: begin an election at term 325 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,229 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,739 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection296: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t325, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t325] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t325, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 344 for changeToCandidate datanode_3 | 2019-10-15 03:16:01,256 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 529 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,739 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 325 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:37,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,085 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection300: begin an election at term 345 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:01,256 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,104 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection300: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t345, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t345] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t345, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,757 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_3 | 2019-10-15 03:16:01,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 530 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:01,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 345 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:38,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:01,370 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,757 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 325 for changeToCandidate datanode_2 | 2019-10-15 03:15:38,184 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 330 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:01,436 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 531 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:37,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,436 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,763 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection297: begin an election at term 326 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:01,436 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:37,773 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection297: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t326, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t326] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t326, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,184 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:01,436 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,773 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 326 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:38,237 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_3 | 2019-10-15 03:16:01,437 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 515 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:38,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,437 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,237 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 345 for changeToCandidate datanode_3 | 2019-10-15 03:16:01,437 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:37,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 341 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:01,437 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,260 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection301: begin an election at term 346 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,466 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 516 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection301: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t346, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t346] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t346, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:01,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 346 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:37,831 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:01,466 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:37,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_3 | 2019-10-15 03:16:01,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 517 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,332 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 331 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,843 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 326 for changeToCandidate datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 532 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,847 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection298: begin an election at term 327 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,332 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,855 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection298: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t327, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t327] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t327, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,365 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_1 | 2019-10-15 03:15:37,855 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 327 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,629 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,365 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 346 for changeToCandidate datanode_3 | 2019-10-15 03:16:01,629 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:01,655 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 518 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,898 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 342 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,396 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection302: begin an election at term 347 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,412 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection302: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t347, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t347] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t347, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:01,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,412 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 347 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:37,899 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:01,655 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,412 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:37,928 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_3 | 2019-10-15 03:16:01,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 519 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,412 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,928 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_1 | 2019-10-15 03:15:37,928 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 327 for changeToCandidate datanode_3 | 2019-10-15 03:16:01,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,928 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:01,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 347 for changeToCandidate datanode_1 | 2019-10-15 03:15:37,936 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection299: begin an election at term 328 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 520 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:37,945 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection299: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t328, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,512 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection303: begin an election at term 348 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:37,963 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection299: begin an election at term 329 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 533 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,514 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 332 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:37,980 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection299: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t329, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t329] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t329, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:37,980 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 329 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:37,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,845 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,514 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:37,980 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,027 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 343 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:01,845 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,524 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection303: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t348, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:38,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:01,997 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 521 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,027 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,540 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection303: begin an election at term 349 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,064 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 344 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,566 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection303: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t349, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t349] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t349, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:01,997 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,566 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 349 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:01,997 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 534 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,064 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:01,998 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,087 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 345 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,632 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_3 | 2019-10-15 03:16:01,998 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,059 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 535 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,632 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 349 for changeToCandidate datanode_1 | 2019-10-15 03:15:38,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:02,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:38,087 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,655 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection304: begin an election at term 350 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:38,168 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:16:02,059 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,657 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 333 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,171 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 536 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,168 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 329 for changeToCandidate datanode_3 | 2019-10-15 03:16:02,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection300: begin an election at term 330 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,657 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,171 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection304: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t350, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t350] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t350, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:38,208 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection300: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t330, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t330] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t330, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,675 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 350 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:38,208 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 330 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:02,203 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 522 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,208 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:38,675 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:02,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,720 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 334 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:02,203 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,208 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,230 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 523 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,261 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 346 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,720 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:02,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:38,762 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_1 | 2019-10-15 03:15:38,262 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,230 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,304 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_3 | 2019-10-15 03:16:02,246 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 524 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:38,763 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 350 for changeToCandidate datanode_1 | 2019-10-15 03:15:38,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,304 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 330 for changeToCandidate datanode_2 | 2019-10-15 03:15:38,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:02,246 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,304 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,776 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection305: begin an election at term 351 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,246 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,330 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection301: begin an election at term 331 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,792 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection305: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t351, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t351] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t351, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,247 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,361 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection301: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t331, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t331] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t331, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 351 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:02,279 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 525 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 331 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:38,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:02,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:38,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 335 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:02,279 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 347 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,295 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 537 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,826 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,398 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,948 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_3 | 2019-10-15 03:16:02,295 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,505 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_2 | 2019-10-15 03:15:38,948 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,452 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 538 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 351 for changeToCandidate datanode_3 | 2019-10-15 03:16:02,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,505 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 331 for changeToCandidate datanode_2 | 2019-10-15 03:15:38,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:02,452 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:38,954 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection306: begin an election at term 352 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:38,512 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection302: begin an election at term 332 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,963 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection306: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t352, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t352] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t352, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,478 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 526 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 348 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:38,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 352 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:02,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:38,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:02,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:38,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,478 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,514 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:38,990 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 336 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:02,480 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 539 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,540 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection302: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t332, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t332] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t332, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:38,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,540 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 332 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:38,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,540 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:38,990 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,480 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,540 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,100 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_1 | 2019-10-15 03:15:38,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 349 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:02,615 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 527 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,615 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,101 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 352 for changeToCandidate datanode_3 | 2019-10-15 03:16:02,616 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,542 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:02,616 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,627 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | 2019-10-15 03:15:39,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 337 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:02,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 528 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:02,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 540 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 332 for changeToCandidate datanode_2 | 2019-10-15 03:15:39,103 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:39,117 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection307: begin an election at term 353 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection303: begin an election at term 333 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,143 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection307: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t353, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t353] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t353, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,678 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 353 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:38,657 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 350 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:02,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:02,678 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:02,887 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 529 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 338 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,657 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,676 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection303: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t333, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t333] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t333, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:02,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,676 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 333 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:02,888 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,144 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,676 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:02,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 541 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,188 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 339 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,704 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_3 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,188 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:02,903 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,704 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 333 for changeToCandidate datanode_2 | 2019-10-15 03:15:39,219 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | 2019-10-15 03:16:03,116 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 530 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,705 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:39,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,718 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection304: begin an election at term 334 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,219 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 353 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,735 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection304: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t334, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t334] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t334, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:03,116 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,735 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 334 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 542 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,232 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection308: begin an election at term 354 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:38,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:03,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,248 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection308: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t354, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t354] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t354, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 351 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,248 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 354 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,119 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:03,220 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 543 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:03,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,778 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,307 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 340 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,819 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:83ms, electionTimeout:83ms datanode_2 | 2019-10-15 03:15:39,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,819 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 334 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,307 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:39,371 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | 2019-10-15 03:16:03,220 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,824 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection305: begin an election at term 335 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,300 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 531 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,831 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection305: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t335, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t335] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t335, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,371 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 354 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 335 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:39,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:03,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:38,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:39,393 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection309: begin an election at term 355 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,300 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,408 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection309: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t355, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t355] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t355, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,456 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 544 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:38,956 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 352 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,408 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 355 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,456 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,408 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:38,956 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,456 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,408 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,956 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:03,456 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,491 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 341 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:38,956 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:03,513 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 532 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,984 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | 2019-10-15 03:16:03,513 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:38,984 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,513 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,491 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:38,984 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 335 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,513 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,501 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_1 | 2019-10-15 03:15:38,984 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:03,563 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 533 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:38,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection306: begin an election at term 336 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,501 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 355 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,005 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection306: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t336, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t336] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t336, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:03,563 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,527 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection310: begin an election at term 356 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,005 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 336 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,586 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 545 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,005 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:39,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection310: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t356, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t356] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t356, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,005 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 356 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,071 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_3 | 2019-10-15 03:16:03,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:39,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 534 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,072 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 336 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,571 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 342 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 546 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,571 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,101 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection307: begin an election at term 337 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,571 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,117 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection307: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t337, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,571 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,118 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 353 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:03,700 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,595 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_1 | 2019-10-15 03:15:39,118 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,700 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,595 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,119 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,595 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 356 for changeToCandidate datanode_3 | 2019-10-15 03:16:03,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 547 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,142 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection307: begin an election at term 338 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,595 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:03,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,159 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection307: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t338, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t338] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t338, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,612 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection311: begin an election at term 357 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:03,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,159 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 338 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,775 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:03,925 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 535 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:03,925 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,628 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection311: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t357, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t357] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t357, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,173 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_3 | 2019-10-15 03:16:03,926 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:03,926 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,173 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 338 for changeToCandidate datanode_1 | 2019-10-15 03:15:39,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:03,979 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 548 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,628 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 357 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:03,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,187 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection308: begin an election at term 339 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:03,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,204 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection308: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t339, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t339] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t339, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:39,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:03,979 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 339 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:39,740 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | 2019-10-15 03:16:04,008 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 536 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:39,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,008 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,740 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 357 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,008 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,234 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 354 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:04,008 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,762 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection312: begin an election at term 358 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,034 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 549 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,772 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection312: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t358, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t358] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t358, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,234 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,772 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 358 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:39,291 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_3 | 2019-10-15 03:16:04,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:39,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:04,034 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,797 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 343 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,292 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 339 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,036 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 537 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:04,036 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,797 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,305 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection309: begin an election at term 340 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,036 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 344 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection309: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t340, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t340] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t340, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,036 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,324 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 340 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:39,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:04,075 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 538 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:39,826 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,867 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_3 | 2019-10-15 03:16:04,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,394 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 355 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,867 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,075 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,394 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,868 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 358 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,198 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 550 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,394 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:04,198 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,395 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,881 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection313: begin an election at term 359 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,198 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,465 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | 2019-10-15 03:15:39,898 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection313: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t359, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t359] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t359, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,198 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,465 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,898 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 359 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:04,266 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 551 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,465 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 340 for changeToCandidate datanode_2 | 2019-10-15 03:15:39,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:04,266 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,465 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:04,266 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,489 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection310: begin an election at term 341 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,267 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,297 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 539 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,505 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection310: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t341, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t341] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t341, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,900 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_1 | 2019-10-15 03:15:39,506 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 341 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:04,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,506 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:39,900 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 359 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,298 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,506 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:04,353 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 540 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 356 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:04,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection314: begin an election at term 360 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,916 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection314: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t360, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:04,354 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,930 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection314: begin an election at term 361 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,529 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 541 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,545 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,945 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection314: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t361, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t361] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t361, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,545 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 552 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,545 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 341 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,945 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 361 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:39,545 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:39,570 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection311: begin an election at term 342 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:39,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,596 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection311: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t342, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t342] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t342, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,406 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,984 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 345 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,596 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 342 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:04,406 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:39,984 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,597 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 553 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:39,984 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,597 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,984 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,613 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 357 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 542 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:39,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 346 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:39,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:39,999 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,613 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,128 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,128 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,587 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 358 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,129 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 361 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:04,587 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,148 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection315: begin an election at term 362 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:04,637 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 543 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:40,165 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection315: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t362, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t362] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t362, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,765 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,637 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,790 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | 2019-10-15 03:16:04,637 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,165 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 362 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:39,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,637 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:39,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 342 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,738 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 554 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:04,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 347 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,796 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection312: begin an election at term 343 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection312: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t343, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t343] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t343, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,738 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 343 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:40,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:04,811 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 544 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:40,270 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_3 | 2019-10-15 03:16:04,811 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:04,811 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,819 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | 2019-10-15 03:15:40,270 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 362 for changeToCandidate datanode_3 | 2019-10-15 03:16:04,811 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:04,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 555 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,819 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 343 for changeToCandidate datanode_2 | 2019-10-15 03:15:40,300 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection316: begin an election at term 363 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,820 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:04,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,308 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection316: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t363, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t363] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t363, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:39,824 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection313: begin an election at term 344 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,863 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,308 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 363 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:39,839 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection313: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t344, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t344] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t344, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:04,989 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 556 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:39,839 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 344 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:04,989 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:39,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:04,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,322 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_1 | 2019-10-15 03:15:39,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:04,990 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,042 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 545 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,883 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 359 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,322 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 363 for changeToCandidate datanode_3 | 2019-10-15 03:16:05,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:05,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,332 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection317: begin an election at term 364 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,333 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 348 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,883 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:05,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 546 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:40,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,916 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 360 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:05,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:16:05,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:05,222 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 557 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:39,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,334 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:05,222 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,916 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:05,222 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,931 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 361 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:05,222 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,334 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:05,263 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 547 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,348 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection317: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t364, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t364] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t364, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,931 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,348 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 364 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:05,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,968 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:15:40,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:05,264 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:05,424 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 548 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:39,969 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 344 for changeToCandidate datanode_2 | 2019-10-15 03:15:40,377 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_3 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:39,969 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:39,982 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection314: begin an election at term 345 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,424 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:39,986 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection314: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t345, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,377 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 364 for changeToCandidate datanode_3 | 2019-10-15 03:16:05,440 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 558 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:39,998 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection314: begin an election at term 346 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,440 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,402 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection318: begin an election at term 365 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,025 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection314: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t346, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t346] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t346, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,440 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 346 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:40,418 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection318: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t365, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t365] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t365, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,440 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:40,418 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 365 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:05,480 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 559 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,418 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:05,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,150 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 362 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,418 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:05,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,516 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_3 | 2019-10-15 03:16:05,480 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,517 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,539 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 560 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,150 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,517 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 365 for changeToCandidate datanode_3 | 2019-10-15 03:16:05,539 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,192 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_2 | 2019-10-15 03:15:40,517 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:05,540 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,192 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,525 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection319: begin an election at term 366 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,540 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,192 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 346 for changeToCandidate datanode_2 | 2019-10-15 03:15:40,529 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 349 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:05,598 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 549 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,192 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:40,529 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,206 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection315: begin an election at term 347 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,529 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:05,599 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 561 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,223 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection315: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t347, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t347] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t347, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,529 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:05,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,223 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 347 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:40,551 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection319: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t366, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t366] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t366, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:05,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 366 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:05,599 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:40,302 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 363 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:40,302 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:05,599 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 350 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,302 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:05,599 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,302 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:40,318 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 550 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:40,551 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 562 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,565 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 351 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,318 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 347 for changeToCandidate datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,565 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,565 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:40,332 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection316: begin an election at term 348 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,565 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,333 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 364 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,599 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_3 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,111 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 366 for changeToCandidate datanode_3 | 2019-10-15 03:16:06,111 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,333 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:06,195 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 563 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection316: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t348, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t348] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t348, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,711 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection320: begin an election at term 367 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,348 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 348 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:06,195 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 352 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:06,195 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,195 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,272 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 564 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,404 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 365 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,713 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,735 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection320: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t367, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t367] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t367, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:06,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,735 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 367 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:06,273 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:06,334 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 551 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,404 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,524 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:15:40,881 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_3 | 2019-10-15 03:16:06,334 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:40,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,524 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,334 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,524 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 348 for changeToCandidate datanode_2 | 2019-10-15 03:15:40,881 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 367 for changeToCandidate datanode_3 | 2019-10-15 03:16:06,405 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 565 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,524 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:40,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:06,405 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,527 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: begin an election at term 349 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,896 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection321: begin an election at term 368 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:06,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,527 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 366 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,911 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection321: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t368, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t368] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t368, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:06,406 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,527 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,911 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 368 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:06,580 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 552 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,527 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:40,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:06,580 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,527 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:40,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,580 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,531 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t349, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,917 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 353 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:06,580 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,550 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: begin an election at term 350 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,641 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 566 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,556 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t350, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,641 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:40,918 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,641 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,563 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: begin an election at term 351 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 354 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:06,641 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,629 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t351, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,670 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 567 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: begin an election at term 352 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:40,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,670 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 367 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:40,964 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,670 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,058 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 355 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:06,670 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,058 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 553 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:41,058 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:06,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,058 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,713 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,108 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_3 | 2019-10-15 03:16:06,774 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,735 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection317: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t352, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t352] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t352, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,843 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 568 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:40,735 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 352 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:06,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 368 for changeToCandidate datanode_3 | 2019-10-15 03:16:06,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:06,843 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:40,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:06,943 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 569 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,126 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection322: begin an election at term 369 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,898 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 368 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:06,943 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,132 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection322: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t369, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t369] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t369, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:06,943 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:40,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:40,898 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:06,943 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,133 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 369 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:40,910 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_3 | 2019-10-15 03:16:07,007 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 554 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,133 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:40,910 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,007 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,133 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:40,910 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 352 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,007 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,238 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 356 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:40,910 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,008 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,238 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,052 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 555 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,238 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:41,238 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:07,052 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,916 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection318: begin an election at term 353 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,309 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 357 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:07,052 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,309 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,053 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,926 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection318: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t353, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t353] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t353, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,309 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:07,104 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 556 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,309 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:40,926 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 353 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:07,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,310 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_1 | 2019-10-15 03:15:40,926 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:07,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,926 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,104 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,310 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 369 for changeToCandidate datanode_1 | 2019-10-15 03:15:40,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_3 | 2019-10-15 03:16:07,155 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 570 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:40,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,155 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,325 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection323: begin an election at term 370 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 353 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,155 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,338 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection323: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t370, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t370] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t370, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:40,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,155 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 370 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:40,962 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection319: begin an election at term 354 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 571 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:40,969 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection319: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t354, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t354] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t354, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:40,970 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 354 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:07,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,379 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_1 | 2019-10-15 03:15:40,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:07,181 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:40,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,197 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 572 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,379 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 370 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,049 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_3 | 2019-10-15 03:16:07,197 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,197 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 354 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,197 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,385 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection324: begin an election at term 371 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,050 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,269 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 557 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,393 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection324: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t371, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection320: begin an election at term 355 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,269 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,410 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection324: begin an election at term 372 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,269 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,065 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection320: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t355, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t355] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t355, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,422 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection324: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t372, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t372] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t372, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,269 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 355 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:41,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 372 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:07,380 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 573 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:41,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:07,380 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:41,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:07,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,128 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 369 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:07,381 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,128 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,423 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 358 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,128 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,408 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 558 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,423 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,128 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:07,408 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,423 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,227 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_3 | 2019-10-15 03:16:07,408 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,423 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,227 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,456 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 359 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:07,408 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,227 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 355 for changeToCandidate datanode_2 | 2019-10-15 03:15:41,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,479 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 574 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,227 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,236 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection321: begin an election at term 356 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,457 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,256 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection321: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t356, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t356] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t356, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,460 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | 2019-10-15 03:16:07,591 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 559 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,591 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,460 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 372 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 356 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:07,592 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,461 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:41,465 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection325: begin an election at term 373 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,592 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,475 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection325: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t373, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t373] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t373, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,301 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_3 | 2019-10-15 03:16:07,695 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 575 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,475 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 373 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:41,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,695 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:41,301 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 356 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,695 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,695 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,557 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:81ms, electionTimeout:81ms datanode_1 | 2019-10-15 03:15:41,307 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection322: begin an election at term 357 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,742 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 560 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,326 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection322: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t357, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t357] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t357, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,557 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 373 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,742 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,327 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 357 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:41,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:41,564 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection326: begin an election at term 374 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,742 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,598 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection326: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t374, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t374] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t374, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,327 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 370 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:07,742 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,598 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 374 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:07,776 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 576 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,598 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:07,776 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,328 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:41,598 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:07,776 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,651 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 360 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,328 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:07,776 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,387 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 371 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:07,812 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 577 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,388 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,812 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,651 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,388 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,812 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,665 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_1 | 2019-10-15 03:15:41,388 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:07,812 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,410 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:83ms, electionTimeout:83ms datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 578 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,665 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 374 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,410 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 561 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection327: begin an election at term 375 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,700 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection327: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t375, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t375] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t375, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,410 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 357 for changeToCandidate datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,700 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 375 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:41,410 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:16:07,870 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:07,870 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,412 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 372 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:07,890 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 579 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 361 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:07,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,412 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:07,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,421 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection323: begin an election at term 358 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:07,890 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,431 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection323: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t358, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t358] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t358, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,038 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 580 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:08,038 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,703 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,431 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 358 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,038 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:08,038 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,827 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | 2019-10-15 03:15:41,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 562 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,440 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_3 | 2019-10-15 03:16:08,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 375 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,440 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 358 for changeToCandidate datanode_3 | 2019-10-15 03:16:08,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:08,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,876 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection328: begin an election at term 376 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,455 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection324: begin an election at term 359 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,161 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 563 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:41,878 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 362 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:08,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,464 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection324: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t359, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t359] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t359, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 359 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,161 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,878 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:08,246 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 581 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection328: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t376, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t376] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t376, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,246 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 376 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:41,467 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 373 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:08,246 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:41,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:41,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,247 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 582 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,951 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_1 | 2019-10-15 03:15:41,467 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,566 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 374 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,951 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,951 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 376 for changeToCandidate datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 564 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:41,951 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,566 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:41,956 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection329: begin an election at term 377 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,294 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,638 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | 2019-10-15 03:15:41,973 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection329: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t377, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t377] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t377, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,638 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:41,973 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 377 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,295 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,638 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 359 for changeToCandidate datanode_2 | 2019-10-15 03:15:41,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:41,638 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:08,431 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 583 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:41,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,649 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection325: begin an election at term 360 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,022 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 363 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection325: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t360, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t360] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t360, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,432 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,674 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 360 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:08,473 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 565 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 375 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:08,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_1 | 2019-10-15 03:15:41,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,473 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,526 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 584 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:42,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 377 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,677 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:08,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,696 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_3 | 2019-10-15 03:16:08,526 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,085 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection330: begin an election at term 378 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,526 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,086 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 364 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:41,697 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 360 for changeToCandidate datanode_3 | 2019-10-15 03:16:08,628 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 566 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:08,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,701 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection326: begin an election at term 361 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,087 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:08,628 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:08,715 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 567 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:08,715 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,104 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection330: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t378, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t378] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t378, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:41,713 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection326: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t361, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t361] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t361, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 378 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,715 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 361 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:42,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:08,715 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:42,105 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:08,750 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 585 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:42,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 365 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:08,750 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,819 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | 2019-10-15 03:15:42,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,750 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:41,819 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 361 for changeToCandidate datanode_3 | 2019-10-15 03:16:08,750 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,158 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,819 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:08,771 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 568 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,250 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_1 | 2019-10-15 03:15:41,876 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection327: begin an election at term 362 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,771 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,878 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 376 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:08,771 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 378 for changeToCandidate datanode_1 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,772 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:41,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:08,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 569 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,265 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection331: begin an election at term 379 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,282 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection331: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t379, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t379] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t379, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,282 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 379 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,815 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,878 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:08,892 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 586 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,899 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection327: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t362, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t362] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t362, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:08,892 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:41,899 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 362 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:42,353 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 366 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:08,892 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:42,353 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,892 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:42,353 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:08,918 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 587 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:41,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 377 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:42,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,918 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,425 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:16:08,918 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:41,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:42,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:08,918 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:41,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,425 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 379 for changeToCandidate datanode_1 | 2019-10-15 03:15:42,016 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | 2019-10-15 03:16:08,986 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 570 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:42,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection332: begin an election at term 380 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,016 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 362 for changeToCandidate datanode_2 | 2019-10-15 03:15:42,465 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection332: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t380, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t380] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t380, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:08,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:42,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 380 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:08,986 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection328: begin an election at term 363 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:09,034 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 571 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,051 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection328: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t363, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t363] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t363, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:09,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,051 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 363 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:42,527 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 367 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:09,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:42,527 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,034 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,527 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:09,103 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 588 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:42,527 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,076 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_3 | 2019-10-15 03:16:09,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,620 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_1 | 2019-10-15 03:15:42,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,620 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,076 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 363 for changeToCandidate datanode_3 | 2019-10-15 03:16:09,104 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,620 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 380 for changeToCandidate datanode_1 | 2019-10-15 03:15:42,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:42,620 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:09,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 572 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection329: begin an election at term 364 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,634 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection333: begin an election at term 381 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,086 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 378 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:09,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:42,650 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection333: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t381, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t381] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t381, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,120 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,087 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:09,149 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 589 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,104 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection329: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t364, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t364] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t364, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,650 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 381 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:42,105 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 364 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:42,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:09,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:42,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:09,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:42,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 368 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 573 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,142 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_2 | 2019-10-15 03:15:42,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,142 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,702 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 590 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,142 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 364 for changeToCandidate datanode_2 | 2019-10-15 03:15:42,702 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:42,779 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection330: begin an election at term 365 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:42,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,779 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 381 for changeToCandidate datanode_1 | 2019-10-15 03:15:42,173 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection330: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t365, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t365] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t365, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,339 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:09,339 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,811 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection334: begin an election at term 382 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 574 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,173 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 365 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:42,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 369 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:09,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:09,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,267 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 379 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:09,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,268 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection334: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t382, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t382] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t382, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,268 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:09,515 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 591 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,268 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 382 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:09,515 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,338 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_2 | 2019-10-15 03:15:42,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:09,515 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 365 for changeToCandidate datanode_3 | 2019-10-15 03:16:09,515 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,935 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_1 | 2019-10-15 03:15:42,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:09,608 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 575 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,351 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection331: begin an election at term 366 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,608 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,936 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 382 for changeToCandidate datanode_1 | 2019-10-15 03:15:42,368 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection331: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t366, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t366] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t366, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,608 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:42,368 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 366 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:09,608 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,950 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection335: begin an election at term 383 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:09,635 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 576 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:42,977 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection335: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t383, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t383] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t383, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:09,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,441 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 380 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:09,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 383 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:42,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,635 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:42,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:09,670 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 592 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,441 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:09,670 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 370 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,510 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:16:09,670 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,670 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:42,978 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,511 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 366 for changeToCandidate datanode_2 | 2019-10-15 03:15:42,978 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:09,751 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 593 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,033 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_3 | 2019-10-15 03:16:09,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,525 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection332: begin an election at term 367 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,033 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,541 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection332: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t367, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t367] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t367, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,751 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,034 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 383 for changeToCandidate datanode_3 | 2019-10-15 03:16:09,823 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 594 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,542 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 367 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:09,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,048 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection336: begin an election at term 384 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:42,636 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 381 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,064 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection336: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t384, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t384] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t384, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,064 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 384 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:42,636 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:09,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:42,686 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | 2019-10-15 03:16:09,823 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,827 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 577 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,088 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_1 | 2019-10-15 03:15:42,686 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 367 for changeToCandidate datanode_2 | 2019-10-15 03:15:43,088 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:09,827 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,088 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 384 for changeToCandidate datanode_3 | 2019-10-15 03:16:09,827 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,699 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection333: begin an election at term 368 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:09,827 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,751 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection333: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t368, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t368] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t368, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,103 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection337: begin an election at term 385 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,752 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 368 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,120 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection337: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t385, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t385] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t385, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:09,924 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 595 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,120 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 385 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:09,924 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:42,788 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_2 | 2019-10-15 03:15:43,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:09,924 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,178 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_1 | 2019-10-15 03:15:42,788 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 368 for changeToCandidate datanode_3 | 2019-10-15 03:16:09,924 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,018 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 578 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,179 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 385 for changeToCandidate datanode_3 | 2019-10-15 03:16:10,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,811 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection334: begin an election at term 369 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:10,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,192 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection338: begin an election at term 386 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,018 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,813 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 382 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,194 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 371 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:10,045 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 579 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,813 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:10,045 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:42,813 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,045 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,827 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection334: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t369, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t369] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t369, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,045 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,210 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection338: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t386, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t386] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t386, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:42,828 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 369 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:10,092 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 596 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,210 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 386 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:42,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:10,093 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:42,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,093 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 383 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,243 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_3 | 2019-10-15 03:16:10,093 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,196 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 580 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,243 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 386 for changeToCandidate datanode_3 | 2019-10-15 03:16:10,196 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,952 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:10,196 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,952 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | 2019-10-15 03:15:43,257 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection339: begin an election at term 387 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,196 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,273 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection339: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t387, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t387] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t387, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,248 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 581 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:42,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 369 for changeToCandidate datanode_2 | 2019-10-15 03:15:43,273 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 387 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:10,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:10,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,976 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection335: begin an election at term 370 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,248 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:42,993 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection335: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t370, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t370] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t370, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 372 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:10,310 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 597 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:42,994 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 370 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,310 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:42,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,310 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:42,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:10,310 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,274 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,352 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 598 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:10,352 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 384 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:10,352 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,293 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_1 | 2019-10-15 03:15:43,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,352 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,383 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 582 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,293 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 387 for changeToCandidate datanode_1 | 2019-10-15 03:15:43,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:10,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,050 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,307 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection340: begin an election at term 388 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 385 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:10,383 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,331 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection340: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t388, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t388] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t388, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 583 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,332 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 388 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:10,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:10,442 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,105 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,177 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_3 | 2019-10-15 03:16:10,457 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 584 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,381 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 373 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,177 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 370 for changeToCandidate datanode_3 | 2019-10-15 03:16:10,457 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:10,457 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,381 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,192 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection336: begin an election at term 371 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 585 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,440 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 374 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 386 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:16:10,564 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 599 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,440 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,194 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,452 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_3 | 2019-10-15 03:16:10,564 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,210 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection336: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t371, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t371] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t371, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,654 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 586 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,211 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 371 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,452 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 388 for changeToCandidate datanode_3 | 2019-10-15 03:16:10,654 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:10,654 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,456 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection341: begin an election at term 389 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,654 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,258 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 387 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection341: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t389, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t389] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t389, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:10,682 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 587 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 389 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:10,682 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:10,682 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,259 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,682 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,262 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | 2019-10-15 03:15:43,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 375 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:10,744 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 588 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,744 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 371 for changeToCandidate datanode_2 | 2019-10-15 03:15:43,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,744 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,550 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,744 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,272 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection337: begin an election at term 372 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,638 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 376 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:10,761 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 600 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:43,288 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection337: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t372, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t372] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t372, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,638 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,761 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 372 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,638 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:10,762 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,638 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:10,762 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,641 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_1 | 2019-10-15 03:15:43,308 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 388 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:10,844 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 601 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,641 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:10,844 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,641 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 389 for changeToCandidate datanode_3 | 2019-10-15 03:16:10,844 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,641 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:10,844 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,309 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,653 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection342: begin an election at term 390 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,295 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 602 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:43,364 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_2 | 2019-10-15 03:15:43,669 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection342: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t390, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t390] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t390, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,295 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 589 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,365 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 372 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,669 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 390 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:43,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,379 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection338: begin an election at term 373 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,296 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection338: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t373, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t373] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t373, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,839 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | 2019-10-15 03:16:11,296 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,396 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 373 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,840 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,296 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,840 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 390 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,355 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 590 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,840 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,415 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_3 | 2019-10-15 03:16:11,356 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,851 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection343: begin an election at term 391 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,415 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,356 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,853 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 377 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,415 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 373 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,356 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 591 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,416 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 603 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,853 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,439 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection339: begin an election at term 374 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,867 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection343: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t391, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t391] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t391, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,376 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,455 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection339: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t374, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t374] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t374, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,868 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 391 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,455 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 374 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:43,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:43,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,376 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,909 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_3 | 2019-10-15 03:16:11,430 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 604 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:43,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 389 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,910 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 391 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,430 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:43,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,458 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,924 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection344: begin an election at term 392 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,430 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,533 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_2 | 2019-10-15 03:15:43,928 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 378 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:11,479 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 592 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,928 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,534 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 374 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:11,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,929 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:11,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,548 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection340: begin an election at term 375 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:43,954 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection344: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t392, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t392] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t392, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,565 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection340: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t375, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t375] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t375, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,514 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 593 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:43,955 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 392 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:43,565 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 375 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:11,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:43,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:11,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:43,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:11,514 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:43,975 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 379 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,628 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_3 | 2019-10-15 03:16:11,516 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 605 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:43,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,628 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:43,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,628 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 375 for changeToCandidate datanode_2 | 2019-10-15 03:15:43,975 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:11,516 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,628 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,108 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | 2019-10-15 03:16:11,657 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 594 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,636 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection341: begin an election at term 376 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:44,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,654 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection341: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t376, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t376] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t376, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,109 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 392 for changeToCandidate datanode_1 | 2019-10-15 03:15:43,654 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 376 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:11,657 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:11,657 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:11,715 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 606 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,124 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection345: begin an election at term 393 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 390 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:11,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 380 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:11,716 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,655 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,823 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 595 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,836 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_2 | 2019-10-15 03:15:44,126 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:11,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:11,823 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 376 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,823 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:11,848 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 607 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection345: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t393, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t393] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t393, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,140 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 393 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:11,849 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,140 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:43,851 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection342: begin an election at term 377 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,849 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 596 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,180 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_1 | 2019-10-15 03:15:43,853 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 391 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:11,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,180 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 393 for changeToCandidate datanode_3 | 2019-10-15 03:16:11,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,853 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:11,903 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,853 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:11,936 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 597 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,183 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 381 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,868 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection342: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t377, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t377] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t377, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:11,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,868 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 377 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:11,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:11,936 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,868 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:44,184 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,868 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,198 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection346: begin an election at term 394 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:43,922 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_2 | 2019-10-15 03:15:44,215 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection346: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t394, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t394] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t394, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,101 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 598 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,215 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 394 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,101 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,922 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 377 for changeToCandidate datanode_2 | 2019-10-15 03:15:44,215 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,101 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:43,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:43,926 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 392 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:12,101 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,216 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,926 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,101 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 608 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,374 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 382 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,926 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,102 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,927 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:12,102 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,927 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection343: begin an election at term 378 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,102 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,374 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:43,954 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection343: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t378, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t378] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t378, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,205 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 609 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,386 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_1 | 2019-10-15 03:15:43,955 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 378 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,386 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 394 for changeToCandidate datanode_1 | 2019-10-15 03:15:43,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:12,205 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:43,960 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_2 | 2019-10-15 03:15:44,402 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection347: begin an election at term 395 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,305 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 599 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,960 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,419 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection347: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t395, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t395] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t395, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,960 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 378 for changeToCandidate datanode_3 | 2019-10-15 03:16:12,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,419 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 395 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:43,960 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:12,305 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:43,973 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection344: begin an election at term 379 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,406 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 600 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,991 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection344: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t379, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t379] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t379, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,420 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 383 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:43,991 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 379 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:43,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,406 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:43,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:12,406 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,420 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,111 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_3 | 2019-10-15 03:16:12,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 610 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,546 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 384 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,111 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,546 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,111 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 379 for changeToCandidate datanode_3 | 2019-10-15 03:16:12,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,111 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,546 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:12,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,124 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection345: begin an election at term 380 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 611 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,125 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 393 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:12,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,546 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,125 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,573 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | 2019-10-15 03:16:12,439 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,125 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,510 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 601 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,125 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,573 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 395 for changeToCandidate datanode_3 | 2019-10-15 03:16:12,510 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection345: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t380, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t380] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t380, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:44,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:12,510 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,140 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 380 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,510 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,140 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:44,597 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection348: begin an election at term 396 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,544 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 602 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,614 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection348: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t396, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t396] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t396, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,168 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_3 | 2019-10-15 03:16:12,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,614 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 396 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:44,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,545 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:44,168 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 380 for changeToCandidate datanode_2 | 2019-10-15 03:15:44,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:12,545 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,643 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 385 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:12,609 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 612 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection346: begin an election at term 381 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:44,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,609 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,199 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection346: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t381, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t381] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t381, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:44,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:12,609 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 381 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,609 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,643 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,672 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 613 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,654 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 394 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:12,672 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 603 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,654 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 396 for changeToCandidate datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:44,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:12,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,659 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection349: begin an election at term 397 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,199 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:12,673 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,676 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection349: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t397, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t397] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t397, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,673 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,335 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:135ms, electionTimeout:135ms datanode_2 | 2019-10-15 03:15:44,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 397 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,673 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 604 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,335 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 381 for changeToCandidate datanode_2 | 2019-10-15 03:15:44,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:12,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,828 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | 2019-10-15 03:16:12,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,372 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection347: begin an election at term 382 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:44,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 397 for changeToCandidate datanode_1 | 2019-10-15 03:15:44,388 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection347: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t382, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t382] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t382, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,741 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 614 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,389 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 382 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:44,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:12,741 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:44,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection350: begin an election at term 398 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,741 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,863 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 386 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:12,741 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,404 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 395 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,804 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 605 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:44,404 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,863 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,404 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,878 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection350: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t398, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t398] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t398, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,411 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | 2019-10-15 03:15:44,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 398 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:12,805 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,805 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 615 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,411 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 382 for changeToCandidate datanode_2 | 2019-10-15 03:15:44,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:12,805 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,906 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 387 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:12,805 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:12,805 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:44,418 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection348: begin an election at term 383 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,963 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 606 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:44,906 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:12,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,435 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection348: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t383, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t383] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t383, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:12,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,435 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 383 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:44,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:12,963 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,931 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_1 | 2019-10-15 03:15:44,515 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:15:44,931 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,024 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 607 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,931 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 398 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,515 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 383 for changeToCandidate datanode_2 | 2019-10-15 03:15:44,932 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:13,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:44,945 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection351: begin an election at term 399 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,544 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection349: begin an election at term 384 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,025 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:44,957 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection351: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t399, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection349: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t384, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t384] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t384, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,042 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 616 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,972 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection351: begin an election at term 400 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 384 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:44,987 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection351: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t400, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t400] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t400, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:44,987 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 400 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:44,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:44,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 396 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:44,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,057 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 617 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,026 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 388 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:13,057 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:45,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,057 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,599 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,057 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,627 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:15:45,026 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,064 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 618 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,173 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_1 | 2019-10-15 03:15:44,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 384 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,628 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:13,064 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 400 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,064 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,641 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection350: begin an election at term 385 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:13,109 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 608 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection350: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t385, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t385] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t385, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,179 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection352: begin an election at term 401 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,109 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,657 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 385 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:45,187 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection352: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t401, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t401] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t401, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,109 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:45,187 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 401 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,109 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,161 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 619 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,661 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 397 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:13,162 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,162 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,661 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,162 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:45,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 389 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:13,267 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 609 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:44,821 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_2 | 2019-10-15 03:15:45,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,821 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 385 for changeToCandidate datanode_2 | 2019-10-15 03:15:45,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:45,214 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 390 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:13,268 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,861 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection351: begin an election at term 386 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,214 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 620 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:44,863 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 398 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,214 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,214 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 610 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 391 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,361 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,863 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:13,361 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:44,878 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection351: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t386, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t386] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t386, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,361 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,879 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 386 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,469 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 621 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,289 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:44,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:45,310 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_1 | 2019-10-15 03:15:44,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,890 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_2 | 2019-10-15 03:15:45,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,469 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,310 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 401 for changeToCandidate datanode_1 | 2019-10-15 03:15:44,890 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 386 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,498 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 611 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:44,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:13,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,315 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection353: begin an election at term 402 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,904 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection352: begin an election at term 387 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:44,921 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection352: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t387, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t387] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t387, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,498 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,322 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection353: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t402, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t402] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t402, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,921 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 387 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,632 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 612 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,322 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 402 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:44,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,632 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:44,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,632 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:44,946 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 399 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:13,632 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,355 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_1 | 2019-10-15 03:15:44,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,687 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 613 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,356 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 402 for changeToCandidate datanode_1 | 2019-10-15 03:15:44,946 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,356 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:44,973 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 400 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:44,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,368 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection354: begin an election at term 403 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,687 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,385 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection354: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t403, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t403] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t403, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:44,973 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:13,688 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 622 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,385 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 403 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,011 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_3 | 2019-10-15 03:16:13,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,385 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:45,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,011 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 387 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,385 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:13,723 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 614 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,487 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_1 | 2019-10-15 03:15:45,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection353: begin an election at term 388 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,050 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection353: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t388, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t388] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t388, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,487 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 403 for changeToCandidate datanode_1 | 2019-10-15 03:15:45,050 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 388 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,723 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,050 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:45,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:13,757 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 623 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,500 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection355: begin an election at term 404 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,050 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,502 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 392 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,181 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 401 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:13,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,758 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 615 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,502 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,181 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:13,775 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,517 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection355: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t404, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t404] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t404, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,202 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | 2019-10-15 03:15:45,518 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 404 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,775 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,518 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,775 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,518 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:13,803 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 616 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,202 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 388 for changeToCandidate datanode_2 | 2019-10-15 03:15:45,581 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_3 | 2019-10-15 03:16:13,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:45,581 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,206 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection354: begin an election at term 389 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,581 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 404 for changeToCandidate datanode_3 | 2019-10-15 03:16:13,803 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,207 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection354: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t389, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,581 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 624 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:45,213 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection354: begin an election at term 390 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,596 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection356: begin an election at term 405 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,220 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection354: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t390, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t390] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t390, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,613 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection356: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t405, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t405] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t405, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 617 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,613 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 405 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 390 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,613 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:45,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,613 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:13,963 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,614 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 393 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,274 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_3 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:13,963 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,274 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 390 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 625 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,614 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:45,758 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | 2019-10-15 03:16:14,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,287 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection355: begin an election at term 391 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,312 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection355: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t391, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t391] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t391, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,006 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,758 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 405 for changeToCandidate datanode_1 | 2019-10-15 03:15:45,312 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 391 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,023 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 618 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:45,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,862 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection357: begin an election at term 406 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:14,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,864 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 394 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:14,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,316 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 402 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,066 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 626 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:45,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:14,066 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:45,865 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,067 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,317 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,892 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection357: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t406, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t406] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t406, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,067 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 403 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,892 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 406 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,132 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 619 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:45,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:45,893 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,370 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,904 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_1 | 2019-10-15 03:15:45,485 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_3 | 2019-10-15 03:16:14,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,132 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,904 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 406 for changeToCandidate datanode_1 | 2019-10-15 03:15:45,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 391 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,170 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 620 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:45,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:14,170 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,935 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection358: begin an election at term 407 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,500 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection356: begin an election at term 392 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:45,937 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 395 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:14,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,502 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 404 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,171 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:14,186 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 627 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,937 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,502 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:14,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,958 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection358: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t407, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t407] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t407, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,502 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:45,958 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 407 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,517 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection356: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t392, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t392] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t392, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,186 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:45,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,230 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 621 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:45,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,517 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 392 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:45,994 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 396 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:45,994 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:45,994 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:14,230 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,597 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 405 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:45,995 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,326 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 622 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,066 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:16:14,326 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,598 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,600 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_2 | 2019-10-15 03:15:46,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,354 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 623 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,600 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,066 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 407 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,600 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 392 for changeToCandidate datanode_2 | 2019-10-15 03:15:46,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:45,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:15:45,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection357: begin an election at term 393 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,629 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection357: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t393, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t393] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t393, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,080 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection359: begin an election at term 408 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 393 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,354 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,095 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection359: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t408, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t408] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t408, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,426 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 628 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,096 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 408 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:14,426 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:45,788 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_3 | 2019-10-15 03:16:14,426 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,426 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,102 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_1 | 2019-10-15 03:15:45,788 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 393 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,550 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 624 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:46,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:14,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,102 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 408 for changeToCandidate datanode_1 | 2019-10-15 03:15:45,862 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection358: begin an election at term 394 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:45,865 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 406 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,115 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection360: begin an election at term 409 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,550 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,138 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection360: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t409, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t409] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t409, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,628 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 625 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,138 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 409 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,865 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,892 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection358: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t394, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t394] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t394, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:46,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:14,628 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,892 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 394 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:46,143 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | 2019-10-15 03:16:14,631 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 629 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:45,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:46,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 409 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:14,631 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,913 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_2 | 2019-10-15 03:15:46,167 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection361: begin an election at term 410 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,711 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 626 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,184 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection361: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t410, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t410] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t410, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,711 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,913 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 394 for changeToCandidate datanode_3 | 2019-10-15 03:16:14,712 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,184 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 410 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:14,712 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,753 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 627 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,935 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection359: begin an election at term 395 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,753 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,184 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,937 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 407 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:14,753 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,186 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 397 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,753 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:14,786 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 630 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,937 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,786 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,186 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:14,786 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,958 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection359: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t395, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t395] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t395, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:46,228 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_3 | 2019-10-15 03:16:14,786 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:45,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 395 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:14,841 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 631 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,228 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 410 for changeToCandidate datanode_1 | 2019-10-15 03:15:45,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:45,959 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:14,841 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:45,969 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | 2019-10-15 03:15:46,254 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection362: begin an election at term 411 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:45,969 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,270 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection362: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t411, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t411] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t411, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,841 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:45,969 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 395 for changeToCandidate datanode_2 | 2019-10-15 03:15:46,271 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 411 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:45,969 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:46,271 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:14,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 628 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:46,271 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:45,993 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection360: begin an election at term 396 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,313 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 398 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,008 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection360: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t396, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t396] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t396, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:14,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:14,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,009 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 396 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:46,314 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,025 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 632 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,314 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:15,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,365 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:94ms, electionTimeout:94ms datanode_1 | 2019-10-15 03:15:46,081 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 408 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:15,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,025 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,365 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 411 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,027 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 629 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:15,027 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,081 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,379 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection363: begin an election at term 412 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,027 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,116 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 409 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,397 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection363: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t412, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t412] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t412, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,027 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,116 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 412 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:15,066 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 630 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,116 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,066 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,116 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,066 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,168 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 410 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,442 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_3 | 2019-10-15 03:16:15,067 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,068 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 633 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:46,169 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 412 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,068 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,169 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:46,179 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | 2019-10-15 03:16:15,069 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection364: begin an election at term 413 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,069 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,457 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 399 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,179 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 396 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,240 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 634 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:46,457 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,185 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection361: begin an election at term 397 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:46,457 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:15,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection361: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t397, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t397] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t397, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:46,484 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection364: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t413, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t413] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t413, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,240 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 397 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:46,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 413 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:15,291 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 631 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:46,485 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,291 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,485 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,291 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,255 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 411 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,668 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_3 | 2019-10-15 03:16:15,291 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,320 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 635 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:46,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:15,320 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,668 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 413 for changeToCandidate datanode_1 | 2019-10-15 03:15:46,256 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:15,320 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:46,288 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:16:15,320 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection365: begin an election at term 414 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,443 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 632 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:46,678 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 400 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,288 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 397 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:15,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,443 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,312 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection362: begin an election at term 398 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,534 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 636 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,678 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,328 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection362: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t398, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t398] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t398, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,692 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection365: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t414, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t414] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t414, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,328 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 398 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:15,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,692 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 414 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:46,328 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,534 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,639 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 633 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:46,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,639 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,869 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | 2019-10-15 03:16:15,639 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,639 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,328 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:46,869 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 414 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,694 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 637 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:46,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 412 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:15,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,884 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection366: begin an election at term 415 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,695 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,886 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 401 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:15,695 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,381 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:15,759 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 638 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:15:46,440 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | 2019-10-15 03:15:46,887 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,902 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection366: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t415, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t415] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t415, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,440 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 398 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,902 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 415 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:46,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:15,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:46,455 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection363: begin an election at term 399 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,456 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 413 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:15,787 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 639 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:46,947 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 402 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,456 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,787 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:46,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,456 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:15,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:46,947 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,456 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:15,788 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:46,947 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection363: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t399, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t399] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t399, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,832 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 634 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:47,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_1 | 2019-10-15 03:15:46,484 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 399 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:15,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:15,833 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:47,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 415 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,833 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,660 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:15:47,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 635 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,045 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection367: begin an election at term 416 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,660 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 399 for changeToCandidate datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 640 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:47,063 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection367: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t416, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t416] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t416, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,063 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 416 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:46,675 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection364: begin an election at term 400 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:15,985 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:46,678 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 414 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,063 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 403 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:15,986 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,678 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:16,017 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 636 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:47,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection364: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t400, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t400] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t400, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,064 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,692 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 400 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:16,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,088 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 404 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:16,018 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,132 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 641 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:47,088 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,873 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_3 | 2019-10-15 03:16:16,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,088 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,132 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,088 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 400 for changeToCandidate datanode_3 | 2019-10-15 03:16:16,133 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,157 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 405 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:16,211 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 642 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:47,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,211 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,884 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection365: begin an election at term 401 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,211 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,886 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 415 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:16,211 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,157 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,237 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 637 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:47,214 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_1 | 2019-10-15 03:15:46,886 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,214 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,886 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,214 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 416 for changeToCandidate datanode_1 | 2019-10-15 03:15:46,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection365: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t401, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t401] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t401, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,237 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,215 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:46,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 401 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:16,357 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 643 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:47,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection368: begin an election at term 417 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:46,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:16,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:47,259 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection368: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t417, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t417] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t417, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,932 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_2 | 2019-10-15 03:15:47,260 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 417 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:16,357 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,932 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:16,439 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: remove FOLLOWER cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7:t643, leader=null, voted=null, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f0,i0, conf=0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_1 | 2019-10-15 03:15:46,932 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 401 for changeToCandidate datanode_2 | 2019-10-15 03:15:47,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:46,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:47,262 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_3 | 2019-10-15 03:16:16,439 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 638 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:46,946 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection366: begin an election at term 402 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:46,962 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection366: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t402, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t402] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t402, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,262 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 417 for changeToCandidate datanode_3 | 2019-10-15 03:16:16,439 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:46,962 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 402 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:47,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:16,439 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:46,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:47,276 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection369: begin an election at term 418 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: shutdown datanode_1 | 2019-10-15 03:15:46,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:47,293 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection369: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t418, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t418] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t418, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,443 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_1 | 2019-10-15 03:15:47,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 416 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:16,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,293 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 418 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:47,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:47,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,443 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,293 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,047 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:16,443 INFO impl.StateMachineUpdater: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-StateMachineUpdater: set stopIndex = -1 datanode_2 | 2019-10-15 03:15:47,335 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 406 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,049 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:16:16,447 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7: closes. applyIndex: -1 datanode_2 | 2019-10-15 03:15:47,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:16,449 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_1 | 2019-10-15 03:15:47,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 402 for changeToCandidate datanode_2 | 2019-10-15 03:15:47,335 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:16,450 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-B866F3D9F4B7-SegmentedRaftLogWorker close() datanode_1 | 2019-10-15 03:15:47,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:47,394 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:16:16,542 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:47,060 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection367: begin an election at term 403 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,394 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:47,064 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection367: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t403, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,394 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 418 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:47,087 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection367: begin an election at term 404 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,394 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:47,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection367: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t404, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t404] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t404, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,418 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection370: begin an election at term 419 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:47,103 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 404 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:47,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection370: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t419, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t419] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t419, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:47,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:47,434 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 419 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:47,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:47,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:47,142 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:47,142 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:47,142 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 404 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:47,491 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_1 | 2019-10-15 03:15:47,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:47,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,155 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection368: begin an election at term 405 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:47,491 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 419 for changeToCandidate datanode_1 | 2019-10-15 03:15:47,171 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection368: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t405, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t405] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t405, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:47,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:47,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 405 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:16,617 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 639 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:47,505 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection371: begin an election at term 420 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,532 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection371: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t420, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t420] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t420, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:47,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:16,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,532 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 420 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:47,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,617 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,532 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:47,244 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 417 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:16,687 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 640 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:47,532 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,566 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 407 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,688 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,244 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:16,688 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:47,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,278 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 418 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:47,566 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,278 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:47,665 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 408 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,278 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:47,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,278 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:47,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,319 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:47,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,665 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:47,319 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 405 for changeToCandidate datanode_2 | 2019-10-15 03:15:47,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:47,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:47,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:47,333 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection369: begin an election at term 406 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 420 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:47,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection369: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t406, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t406] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t406, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:47,349 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 406 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:16,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:47,696 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection372: begin an election at term 421 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:16,829 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 641 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,829 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,713 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection372: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t421, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t421] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t421, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:47,420 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 419 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:16,829 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:47,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 421 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:47,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,829 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:47,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:16,909 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:47,420 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:47,756 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:43ms, electionTimeout:43ms datanode_1 | 2019-10-15 03:15:47,507 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 420 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:47,756 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:47,756 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 421 for changeToCandidate datanode_1 | 2019-10-15 03:15:47,507 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:47,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:47,507 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:47,757 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 409 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,540 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:47,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,540 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:47,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,540 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 406 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:47,757 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:47,777 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection373: begin an election at term 422 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:47,564 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection370: begin an election at term 407 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,794 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection373: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t422, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t422] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t422, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:47,794 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 422 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:47,581 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection370: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t407, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t407] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t407, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:47,581 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 407 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:47,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:47,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:47,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:47,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:47,840 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 410 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:47,649 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | 2019-10-15 03:15:47,840 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:47,649 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:47,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:16,952 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:47,649 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 407 for changeToCandidate datanode_2 | 2019-10-15 03:15:47,841 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:47,649 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:47,872 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 411 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:47,663 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection371: begin an election at term 408 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:47,872 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:47,679 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection371: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t408, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t408] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t408, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:47,679 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 408 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:47,872 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:47,872 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,680 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:47,957 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_1 | 2019-10-15 03:15:47,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 421 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:47,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:47,957 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 422 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:47,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:47,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:47,972 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection374: begin an election at term 423 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:47,698 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:47,988 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection374: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t423, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t423] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t423, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:47,741 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_2 | 2019-10-15 03:15:47,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 423 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:47,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:47,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:47,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,741 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 408 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:48,069 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 412 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:48,069 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:16,981 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:47,755 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection372: begin an election at term 409 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,069 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:47,778 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection372: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t409, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t409] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t409, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,069 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 409 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:48,138 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 413 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:48,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:48,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:47,779 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 422 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:48,138 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:47,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:48,173 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:47,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:48,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,779 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 423 for changeToCandidate datanode_1 | 2019-10-15 03:15:47,825 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:48,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:47,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:47,825 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 409 for changeToCandidate datanode_2 | 2019-10-15 03:15:48,175 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 414 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:47,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:48,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:47,839 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection373: begin an election at term 410 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:48,175 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:47,855 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection373: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t410, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t410] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t410, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:48,188 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection375: begin an election at term 424 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:47,855 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 410 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:48,204 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection375: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t424, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t424] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t424, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:17,049 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 642 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:48,204 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 424 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:17,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:48,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:17,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:48,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:15:48,206 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_3 | 2019-10-15 03:16:17,049 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:17,103 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 643 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:47,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:47,857 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_3 | 2019-10-15 03:16:17,103 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:47,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,103 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:48,206 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,173 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:47,857 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 410 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:47,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:48,206 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 424 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:47,871 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection374: begin an election at term 411 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,206 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:47,896 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection374: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t411, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t411] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t411, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,220 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection376: begin an election at term 425 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:47,896 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 411 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:48,237 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection376: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t425, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t425] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t425, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:47,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:48,237 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 425 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:47,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:48,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:47,974 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 423 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:48,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:47,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:47,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_1 | 2019-10-15 03:15:47,974 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:48,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,053 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:48,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 425 for changeToCandidate datanode_1 | 2019-10-15 03:15:48,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:48,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:48,053 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 411 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,360 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection377: begin an election at term 426 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:48,067 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection375: begin an election at term 412 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:48,377 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection377: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t426, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t426] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t426, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,081 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection375: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t412, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,377 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 426 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:48,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection375: begin an election at term 413 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:48,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:48,152 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection375: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t413, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t413] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t413, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:17,309 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:48,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,152 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 413 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:48,404 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 415 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:48,152 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:48,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:48,152 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:48,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,160 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:7ms, electionTimeout:7ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:48,405 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,160 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:48,468 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_1 | 2019-10-15 03:15:48,160 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 413 for changeToCandidate datanode_2 | 2019-10-15 03:15:48,468 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,160 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:48,173 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection376: begin an election at term 414 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,468 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 426 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:48,189 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection376: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t414, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t414] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t414, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,468 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:48,189 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 414 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,482 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection378: begin an election at term 427 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,189 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:48,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 416 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:48,189 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:48,484 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,190 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 424 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:48,498 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection378: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t427, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t427] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t427, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:48,498 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 427 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:48,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,321 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 644 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:48,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:48,190 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:17,321 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:48,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,222 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 425 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:17,321 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:48,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 417 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:48,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,321 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:48,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,368 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:48,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,222 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:48,362 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 426 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:48,499 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:48,568 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_1 | 2019-10-15 03:15:48,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:48,568 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,362 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:48,568 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 427 for changeToCandidate datanode_1 | 2019-10-15 03:15:48,389 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:48,568 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:48,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:48,582 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection379: begin an election at term 428 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,389 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 414 for changeToCandidate datanode_2 | 2019-10-15 03:15:48,635 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection379: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t428, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t428] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t428, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 428 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:48,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection377: begin an election at term 415 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:48,419 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection377: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t415, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t415] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t415, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:48,419 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 415 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:48,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 418 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:48,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,474 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:48,636 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,474 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:48,636 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,474 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 415 for changeToCandidate datanode_3 | 2019-10-15 03:16:17,448 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:48,789 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:48,474 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:48,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:48,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection378: begin an election at term 416 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 428 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:48,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection378: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t416, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:48,484 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 427 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:48,814 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection380: begin an election at term 429 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:48,819 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 419 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:48,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,484 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:48,819 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,497 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection378: begin an election at term 417 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:48,831 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection380: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t429, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t429] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t429, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:48,514 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection378: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t417, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t417] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t417, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,832 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 429 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:48,514 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 417 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:48,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:48,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:48,888 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_1 | 2019-10-15 03:15:48,584 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 428 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:48,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:48,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,556 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 645 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:48,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:48,888 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 429 for changeToCandidate datanode_3 | 2019-10-15 03:16:17,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,584 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:17,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:48,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:48,590 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_3 | 2019-10-15 03:16:17,556 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:48,902 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection381: begin an election at term 430 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,590 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,559 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:48,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection381: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t430, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t430] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t430, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,591 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 417 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:48,919 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 430 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:48,591 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:48,919 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:48,634 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection379: begin an election at term 418 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,919 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,652 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection379: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t418, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t418] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t418, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:48,938 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 420 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:48,652 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 418 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:48,652 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:48,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,652 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:48,809 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:48,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,809 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:48,938 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,809 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 418 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:48,967 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_1 | 2019-10-15 03:15:48,810 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:48,967 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:48,816 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 429 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:48,967 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 430 for changeToCandidate datanode_1 | 2019-10-15 03:15:48,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:48,967 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:48,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:48,981 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection382: begin an election at term 431 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,816 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:48,988 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection382: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t431, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t431] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t431, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:48,816 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection380: begin an election at term 419 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 431 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:48,832 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection380: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t419, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t419] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t419, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:48,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:17,615 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:48,832 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 419 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:48,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:48,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:49,015 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:48,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:49,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:48,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 430 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:49,016 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 431 for changeToCandidate datanode_1 | 2019-10-15 03:15:48,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:49,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:48,904 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:49,021 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection383: begin an election at term 432 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,904 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:49,029 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection383: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t432, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t432] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t432, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:48,923 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:49,029 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 432 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:48,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:49,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:48,923 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 419 for changeToCandidate datanode_2 | 2019-10-15 03:15:49,029 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:48,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:49,030 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 421 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:48,936 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection381: begin an election at term 420 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:48,953 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection381: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t420, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t420] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t420, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:48,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 420 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:49,030 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:49,072 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 422 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:48,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:49,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,743 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 646 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:49,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:48,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,743 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,072 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,982 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 431 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:49,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_3 | 2019-10-15 03:16:17,743 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:48,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:17,743 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:48,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,821 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:49,229 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 432 for changeToCandidate datanode_1 | 2019-10-15 03:15:48,982 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:49,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 432 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:49,232 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 423 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:49,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:49,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,023 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:49,233 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,023 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:49,233 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection384: begin an election at term 433 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,249 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection384: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t433, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t433] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t433, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 420 for changeToCandidate datanode_2 | 2019-10-15 03:15:49,249 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 433 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:49,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:49,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:49,028 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection382: begin an election at term 421 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:49,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,035 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection382: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t421, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t421] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t421, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:49,379 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:49,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 421 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:49,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:49,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:49,380 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 433 for changeToCandidate datanode_1 | 2019-10-15 03:15:49,036 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:49,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:17,934 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:49,058 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | 2019-10-15 03:15:49,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 424 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:49,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:49,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 421 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:49,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:49,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:49,421 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection385: begin an election at term 434 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:49,070 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection383: begin an election at term 422 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection385: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t434, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:49,079 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection383: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t422, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t422] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t422, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,458 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection385: begin an election at term 435 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:49,079 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 422 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:49,486 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection385: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t435, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t435] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t435, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:49,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:49,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:49,486 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 435 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:49,216 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:49,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:49,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:49,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,216 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 422 for changeToCandidate datanode_2 | 2019-10-15 03:15:49,537 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 425 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:49,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:49,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,231 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection384: begin an election at term 423 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:49,537 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 433 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:49,622 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:135ms, electionTimeout:135ms datanode_1 | 2019-10-15 03:15:49,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:49,622 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:17,959 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 647 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:49,623 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 435 for changeToCandidate datanode_1 | 2019-10-15 03:15:49,235 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:17,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:17,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:49,249 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection384: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t423, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t423] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t423, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,648 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection386: begin an election at term 436 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:17,960 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:49,666 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection386: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t436, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t436] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t436, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,249 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 423 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:18,092 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 648 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:49,666 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 436 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:18,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:49,666 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:18,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:49,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:49,666 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:18,092 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,360 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | 2019-10-15 03:15:49,688 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_3 | 2019-10-15 03:16:18,173 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:49,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:49,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 423 for changeToCandidate datanode_2 | 2019-10-15 03:15:49,688 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 436 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:49,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:49,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:49,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection385: begin an election at term 424 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,699 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection387: begin an election at term 437 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:49,422 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection385: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t424, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t424] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t424, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 426 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:49,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 424 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:49,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:49,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:49,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,423 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 434 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:49,701 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:49,716 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection387: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t437, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t437] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t437, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,424 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:49,716 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 437 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:49,424 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:49,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:49,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 435 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:49,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:49,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:49,722 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_1 | 2019-10-15 03:15:49,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:49,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,460 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:49,722 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 437 for changeToCandidate datanode_1 | 2019-10-15 03:15:49,521 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_3 | 2019-10-15 03:16:18,236 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 649 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:49,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:49,521 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:18,236 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,746 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection388: begin an election at term 438 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,521 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 424 for changeToCandidate datanode_3 | 2019-10-15 03:16:18,236 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:49,763 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection388: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t438, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t438] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t438, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,521 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:18,236 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:49,763 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 438 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:18,310 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 650 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:49,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection386: begin an election at term 425 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:18,310 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection386: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t425, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t425] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t425, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 425 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:18,310 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:49,857 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 427 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:18,310 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:49,857 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:18,336 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:49,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:49,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:49,650 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 436 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:49,858 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:49,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,928 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:49,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:49,928 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:49,650 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:49,928 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 438 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:49,685 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | 2019-10-15 03:15:49,928 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:49,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:49,949 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection389: begin an election at term 439 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:49,685 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 425 for changeToCandidate datanode_2 | 2019-10-15 03:15:49,966 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection389: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t439, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t439] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t439, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:49,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:49,966 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 439 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:49,699 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection387: begin an election at term 426 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:49,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:49,701 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 437 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:49,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:50,063 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 428 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:49,701 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:50,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:50,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:50,063 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:49,701 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 429 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:50,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,701 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:18,442 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:50,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:49,716 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection387: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t426, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t426] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t426, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:50,079 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:49,716 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 426 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:49,716 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:49,716 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:15:49,748 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 438 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:50,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 430 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:49,748 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:49,748 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:49,748 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:49,841 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms datanode_1 | 2019-10-15 03:15:49,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:50,164 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_2 | 2019-10-15 03:15:50,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:49,841 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 426 for changeToCandidate datanode_2 | 2019-10-15 03:15:50,164 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 439 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:49,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:50,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:49,855 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection388: begin an election at term 427 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,178 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection390: begin an election at term 440 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,871 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection388: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t427, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t427] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t427, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:50,207 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection390: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t440, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t440] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t440, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:49,871 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 427 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:50,207 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 440 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:49,871 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:50,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:49,871 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:49,951 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 439 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:50,333 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 431 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:18,443 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 651 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:49,951 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,334 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:18,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:49,951 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,334 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:18,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:49,951 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,334 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:18,443 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,048 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_2 | 2019-10-15 03:15:50,345 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | 2019-10-15 03:16:18,587 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:50,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,345 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:50,048 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 427 for changeToCandidate datanode_2 | 2019-10-15 03:15:50,345 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 440 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:50,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:50,345 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:50,062 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection389: begin an election at term 428 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,359 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection391: begin an election at term 441 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,070 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection389: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t428, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:50,376 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection391: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t441, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t441] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t441, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,077 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection389: begin an election at term 429 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:50,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 441 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:50,093 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection389: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t429, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t429] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t429, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:50,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:50,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 429 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:50,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:50,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:50,388 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:50,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:50,106 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | 2019-10-15 03:15:50,388 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 441 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:50,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:50,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:50,106 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 429 for changeToCandidate datanode_2 | 2019-10-15 03:15:50,401 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection392: begin an election at term 442 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:50,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:50,416 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection392: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t442, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t442] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t442, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:18,674 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 652 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection390: begin an election at term 430 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,416 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 442 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:18,674 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,137 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection390: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t430, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t430] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t430, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:18,674 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:50,137 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 430 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:50,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:18,674 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:50,447 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | 2019-10-15 03:16:18,807 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:50,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:50,180 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 440 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:50,447 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 442 for changeToCandidate datanode_1 | 2019-10-15 03:15:50,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:50,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:50,180 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:50,306 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | 2019-10-15 03:15:50,469 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection393: begin an election at term 443 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:50,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,471 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 432 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:50,306 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 430 for changeToCandidate datanode_2 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:50,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:50,331 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection391: begin an election at term 431 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,471 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:50,361 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 441 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:50,486 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection393: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t443, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t443] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t443, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:50,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,486 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 443 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:50,486 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:50,361 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:50,486 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:50,361 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,505 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 433 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:50,361 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection391: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t431, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t431] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t431, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:50,505 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:50,362 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 431 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:50,505 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:50,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:50,505 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:50,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,674 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | 2019-10-15 03:16:18,809 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 653 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,402 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 442 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:50,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:18,809 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,402 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,674 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 443 for changeToCandidate datanode_3 | 2019-10-15 03:16:18,809 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:50,402 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:18,809 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,402 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection394: begin an election at term 444 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:18,885 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:50,452 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_2 | 2019-10-15 03:15:50,690 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 434 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:50,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,690 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,452 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 431 for changeToCandidate datanode_2 | 2019-10-15 03:15:50,690 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:50,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:50,690 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,469 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection392: begin an election at term 432 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:50,707 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection394: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t444, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t444] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t444, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,471 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 443 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:50,707 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 444 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:50,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:50,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:50,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:50,471 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:50,855 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 435 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:50,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,485 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection392: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t432, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t432] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t432, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:50,856 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:50,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 432 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:50,856 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:50,881 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_1 | 2019-10-15 03:15:50,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:50,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:50,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 444 for changeToCandidate datanode_1 | 2019-10-15 03:15:50,490 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:50,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:50,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:50,913 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection395: begin an election at term 445 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,490 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 432 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:50,933 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection395: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t445, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t445] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t445, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:18,953 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:50,933 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 445 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:50,503 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection393: begin an election at term 433 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:50,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:50,520 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection393: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t433, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t433] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t433, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:50,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:50,520 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 433 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:50,982 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 436 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:50,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:50,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:50,683 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:50,982 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,683 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:51,129 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_1 | 2019-10-15 03:15:50,683 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 433 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:51,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,683 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:51,129 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 445 for changeToCandidate datanode_1 | 2019-10-15 03:15:50,688 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection394: begin an election at term 434 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:51,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:50,690 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 444 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:51,143 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection396: begin an election at term 446 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:51,155 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection396: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t446, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t446] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t446, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:50,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:51,155 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 446 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:50,690 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:51,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:50,707 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection394: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t434, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t434] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t434, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:51,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:50,707 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 434 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:51,164 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_3 | 2019-10-15 03:16:19,048 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 654 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:51,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,048 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,164 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 446 for changeToCandidate datanode_3 | 2019-10-15 03:16:19,048 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:50,838 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_2 | 2019-10-15 03:15:51,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:19,048 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,838 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,174 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection397: begin an election at term 447 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:19,089 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:50,839 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 434 for changeToCandidate datanode_2 | 2019-10-15 03:15:51,190 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection397: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t447, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t447] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t447, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:50,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:51,190 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 447 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:50,853 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection395: begin an election at term 435 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:50,870 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection395: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t435, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t435] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t435, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:50,870 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 435 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:51,209 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 437 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:50,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:51,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:50,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:50,915 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 445 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,209 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:50,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:50,916 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:50,965 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | 2019-10-15 03:15:51,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 447 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:50,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:50,966 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 435 for changeToCandidate datanode_2 | 2019-10-15 03:15:51,330 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection398: begin an election at term 448 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:50,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:51,331 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 438 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,980 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection396: begin an election at term 436 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:51,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:50,997 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection396: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t436, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t436] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t436, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:51,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:50,997 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 436 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:51,331 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:50,997 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:19,203 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 655 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:50,998 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:19,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,145 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 446 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:19,203 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:51,332 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection398: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t448, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,145 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,203 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,345 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection398: begin an election at term 449 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,145 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:19,263 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:51,361 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection398: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t449, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t449] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t449, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,145 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:51,361 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 449 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:51,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 447 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,361 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:51,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,361 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:51,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,428 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_1 | 2019-10-15 03:15:51,176 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:51,194 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:51,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,428 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:51,194 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 436 for changeToCandidate datanode_2 | 2019-10-15 03:15:51,428 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 449 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:51,428 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:51,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:51,452 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection399: begin an election at term 450 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:51,207 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection397: begin an election at term 437 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,468 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection399: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t450, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t450] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t450, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,224 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection397: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t437, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t437] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t437, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,468 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 450 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:51,224 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 437 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:51,468 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:51,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:51,468 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:51,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,528 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 439 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:51,303 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_2 | 2019-10-15 03:15:51,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:51,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:19,263 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 656 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:51,528 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,303 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,621 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_1 | 2019-10-15 03:15:51,303 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 437 for changeToCandidate datanode_2 | 2019-10-15 03:15:51,621 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:51,303 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:19,263 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,621 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 450 for changeToCandidate datanode_1 | 2019-10-15 03:15:51,328 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection398: begin an election at term 438 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:19,370 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:51,621 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:51,332 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 448 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:51,645 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection400: begin an election at term 451 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:51,662 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection400: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t451, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t451] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t451, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:51,662 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 451 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:51,332 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:51,346 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection398: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t438, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t438] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t438, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:51,346 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 438 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:51,717 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:51,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:51,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:51,717 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 451 for changeToCandidate datanode_1 | 2019-10-15 03:15:51,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:51,347 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 449 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:51,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 440 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:51,347 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:51,347 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:51,454 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 450 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,721 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:51,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,721 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection401: begin an election at term 452 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,454 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:19,416 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 657 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:51,736 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection401: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t452, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t452] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t452, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_3 | 2019-10-15 03:16:19,416 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,736 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 452 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:51,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,416 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:51,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:51,512 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 438 for changeToCandidate datanode_3 | 2019-10-15 03:16:19,416 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:51,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:19,544 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:51,798 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 441 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:51,526 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection399: begin an election at term 439 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:51,798 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,543 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection399: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t439, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t439] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t439, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:51,798 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:51,543 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 439 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:51,798 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:51,543 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:51,830 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 442 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:51,543 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:51,647 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 451 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:51,647 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,830 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:51,647 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,876 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:51,647 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:51,705 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | 2019-10-15 03:15:51,876 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 452 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:51,705 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:51,705 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 439 for changeToCandidate datanode_2 | 2019-10-15 03:15:51,901 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection402: begin an election at term 453 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:51,705 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:51,917 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection402: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t453, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t453] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t453, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:51,719 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection400: begin an election at term 440 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,917 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 453 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:51,723 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 452 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:51,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:51,723 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:51,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:19,544 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 658 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:51,723 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:51,982 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 443 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:19,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,723 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:51,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:51,736 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection400: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t440, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t440] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t440, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:51,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:19,544 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,736 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 440 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:51,983 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:19,577 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 659 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,012 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 444 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:51,736 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:19,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,736 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:19,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:52,013 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:19,578 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,782 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | 2019-10-15 03:15:52,013 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:19,782 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:51,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,114 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:51,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 440 for changeToCandidate datanode_2 | 2019-10-15 03:15:52,114 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:51,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:52,114 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 453 for changeToCandidate datanode_1 | 2019-10-15 03:15:51,796 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection401: begin an election at term 441 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:52,114 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:52,147 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection403: begin an election at term 454 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,803 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection401: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t441, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t441] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t441, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:52,149 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 445 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:51,803 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 441 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:52,149 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,803 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:52,149 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:51,803 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:52,149 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:51,824 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_1 | 2019-10-15 03:15:51,824 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,164 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection403: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t454, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t454] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t454, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:51,824 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 441 for changeToCandidate datanode_2 | 2019-10-15 03:15:52,164 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 454 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:51,824 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:52,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:51,828 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection402: begin an election at term 442 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,164 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:51,867 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection402: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t442, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t442] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t442, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,329 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_1 | 2019-10-15 03:15:51,867 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 442 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:52,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:52,330 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 454 for changeToCandidate datanode_1 | 2019-10-15 03:15:51,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:51,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 453 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:52,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:51,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,355 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection404: begin an election at term 455 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:51,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:19,785 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 660 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,373 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection404: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t455, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t455] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t455, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:51,903 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:19,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:51,968 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:16:19,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:52,373 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 455 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:51,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,785 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,968 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 442 for changeToCandidate datanode_3 | 2019-10-15 03:16:19,888 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 661 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:51,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:19,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:51,981 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection403: begin an election at term 443 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:19,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:52,374 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 446 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:19,888 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:51,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection403: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t443, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,011 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection403: begin an election at term 444 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:19,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 662 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,374 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,027 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection403: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t444, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t444] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t444, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:19,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,374 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,027 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 444 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:19,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:52,473 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_1 | 2019-10-15 03:15:52,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:52,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:19,980 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:52,473 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 455 for changeToCandidate datanode_3 | 2019-10-15 03:16:19,987 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:52,065 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:52,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:52,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:52,480 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection405: begin an election at term 456 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 444 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:52,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:52,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection405: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t456, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t456] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t456, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:52,498 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 456 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:52,147 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection404: begin an election at term 445 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:52,149 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 454 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:52,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:52,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,515 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 447 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:52,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:52,515 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:52,149 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:52,515 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:52,164 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection404: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t445, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t445] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t445, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,515 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:52,553 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 448 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,164 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 445 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:52,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,164 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:52,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:52,553 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,165 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:52,643 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_1 | 2019-10-15 03:15:52,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 455 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:52,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,002 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 663 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,643 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 456 for changeToCandidate datanode_1 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:52,358 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,003 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:52,358 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | 2019-10-15 03:16:20,003 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,667 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection406: begin an election at term 457 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,077 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 664 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,358 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 445 for changeToCandidate datanode_2 | 2019-10-15 03:15:52,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection406: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t457, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t457] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t457, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,077 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:52,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 457 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:20,077 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:52,372 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection405: begin an election at term 446 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,077 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:52,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:52,389 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection405: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t446, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t446] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t446, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:52,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,389 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 446 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:20,129 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 665 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:52,783 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 449 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:52,783 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,130 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,783 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 456 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:20,130 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:52,783 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,130 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,822 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_3 | 2019-10-15 03:16:20,158 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:52,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:52,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:52,483 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:52,822 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 457 for changeToCandidate datanode_1 | 2019-10-15 03:15:52,500 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:52,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:52,500 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:52,835 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection407: begin an election at term 458 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,500 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 446 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:52,844 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection407: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t458, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,500 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:52,863 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection407: begin an election at term 459 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,513 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection406: begin an election at term 447 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:52,864 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 450 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,530 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection406: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t447, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t447] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t447, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:52,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,530 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 447 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:52,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:52,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:52,864 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,538 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:15:52,880 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection407: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t459, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t459] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t459, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,538 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:52,881 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 459 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:52,538 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 447 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:52,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:52,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,538 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:52,889 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_1 | 2019-10-15 03:15:52,552 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection407: begin an election at term 448 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:52,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:52,568 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection407: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t448, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t448] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t448, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:52,568 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 448 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:52,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 459 for changeToCandidate datanode_1 | 2019-10-15 03:15:52,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:20,190 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:52,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:52,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:52,904 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection408: begin an election at term 460 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 457 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:52,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection408: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t460, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t460] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t460, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:52,914 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 460 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:52,914 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:52,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:52,915 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:52,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,015 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 451 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,670 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:52,765 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | 2019-10-15 03:15:53,015 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:53,015 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:52,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,068 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:52,766 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 448 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,068 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:52,766 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:53,068 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 460 for changeToCandidate datanode_1 | 2019-10-15 03:15:52,781 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection408: begin an election at term 449 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:52,798 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection408: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t449, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t449] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t449, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:53,068 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:52,798 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 449 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:53,073 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection409: begin an election at term 461 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:52,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,074 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 452 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:52,836 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_2 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,322 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:52,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:52,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 449 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,075 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:52,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:53,090 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection409: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t461, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t461] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t461, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:52,837 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 458 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:53,090 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 461 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:52,837 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:52,838 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:52,838 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,220 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:52,863 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection409: begin an election at term 450 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:52,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 459 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:53,220 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 461 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:52,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:52,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,234 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection410: begin an election at term 462 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:52,864 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:53,251 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection410: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t462, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t462] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t462, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:52,880 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection409: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t450, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t450] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t450, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:53,251 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 462 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:52,880 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 450 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:53,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:52,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:52,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:53,267 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 453 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:52,906 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 460 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:20,324 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 666 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:53,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,324 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:52,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,324 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:53,267 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:52,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,324 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,280 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_1 | 2019-10-15 03:15:52,906 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,349 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 667 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:53,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,007 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_3 | 2019-10-15 03:16:20,349 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,280 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 462 for changeToCandidate datanode_1 | 2019-10-15 03:15:53,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:53,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 450 for changeToCandidate datanode_3 | 2019-10-15 03:16:20,349 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:53,309 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection411: begin an election at term 463 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:20,349 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,338 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection411: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t463, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t463] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t463, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection410: begin an election at term 451 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,394 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 668 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:53,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 463 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:53,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection410: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t451, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t451] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t451, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,394 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:53,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 451 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:20,394 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:53,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:53,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:20,395 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_1 | 2019-10-15 03:15:53,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,501 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:53,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,058 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:53,351 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 463 for changeToCandidate datanode_1 | 2019-10-15 03:15:53,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:53,351 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:53,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 451 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:53,364 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection412: begin an election at term 464 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:53,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection411: begin an election at term 452 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:53,380 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection412: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t464, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t464] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t464, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,075 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 461 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,381 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 464 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:53,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:53,075 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:53,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection411: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t452, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t452] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t452, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,461 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 454 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:53,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 452 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:53,461 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:53,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:53,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:53,462 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:53,237 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 462 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:53,536 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 455 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:53,237 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:53,252 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_2 | 2019-10-15 03:15:53,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:53,252 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,536 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,520 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:53,252 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 452 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,545 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:53,252 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:53,265 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection412: begin an election at term 453 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,545 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:53,308 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection412: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t453, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t453] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t453, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,545 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 464 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:53,308 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 453 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:53,545 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:53,551 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection413: begin an election at term 465 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:53,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:53,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 456 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:53,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:53,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:53,311 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 463 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:53,555 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:53,311 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:53,557 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection413: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t465, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t465] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t465, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,311 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:53,557 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 465 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:53,311 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:53,366 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 464 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:53,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:53,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,697 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 457 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:53,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:53,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:20,551 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 669 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,366 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:53,697 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,551 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_2 | 2019-10-15 03:15:53,712 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 458 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:20,551 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:53,712 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:53,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:53,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 453 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,713 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:53,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:53,735 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_1 | 2019-10-15 03:15:53,460 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection413: begin an election at term 454 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:53,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection413: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t454, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t454] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t454, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,735 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 465 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:53,476 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 454 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:53,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:53,476 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:53,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection414: begin an election at term 466 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:53,476 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,765 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection414: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t466, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t466] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t466, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:53,523 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:53,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 466 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:53,523 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 454 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:53,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:53,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:53,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection414: begin an election at term 455 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,882 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:53,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,548 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection414: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t455, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:53,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 466 for changeToCandidate datanode_1 | 2019-10-15 03:15:53,553 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 465 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:53,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:53,553 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,896 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection415: begin an election at term 467 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,551 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:53,553 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,551 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:53,553 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,599 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:53,898 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 459 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,553 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection414: begin an election at term 456 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection414: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t456, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t456] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t456, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:53,601 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 456 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:53,898 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:53,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:53,913 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection415: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t467, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t467] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t467, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:53,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:53,913 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 467 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:53,668 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | 2019-10-15 03:15:53,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:53,668 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:53,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:53,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 456 for changeToCandidate datanode_2 | 2019-10-15 03:15:53,938 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:53,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:53,939 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:53,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection415: begin an election at term 457 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,939 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 467 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:53,707 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection415: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t457, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,939 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:53,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection415: begin an election at term 458 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,952 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection416: begin an election at term 468 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:53,726 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection415: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t458, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t458] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t458, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:53,969 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection416: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t468, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t468] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t468, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:53,726 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 458 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:53,969 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 468 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:53,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:53,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:53,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:53,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:53,750 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 466 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:20,650 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 670 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:54,020 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 460 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,751 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,020 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,751 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,650 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:54,020 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:20,650 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:53,751 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,020 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,736 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 671 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,891 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_2 | 2019-10-15 03:15:54,076 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:16:20,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:20,736 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:54,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 468 for changeToCandidate datanode_1 | 2019-10-15 03:15:53,891 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 458 for changeToCandidate datanode_2 | 2019-10-15 03:15:54,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:53,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:20,736 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection417: begin an election at term 469 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,896 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection416: begin an election at term 459 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,109 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection417: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t469, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t469] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t469, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,898 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 467 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:54,109 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 469 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:20,763 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:54,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:53,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:53,898 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:54,135 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 461 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:53,913 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection416: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t459, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t459] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t459, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:54,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:53,913 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 459 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:54,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:53,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:54,136 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:53,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:53,954 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 468 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:54,211 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_1 | 2019-10-15 03:15:53,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:53,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,211 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 469 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:53,954 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:53,993 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_2 | 2019-10-15 03:15:54,226 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection418: begin an election at term 470 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:53,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,248 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection418: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t470, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,994 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 459 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:54,274 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection418: begin an election at term 471 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:53,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:54,279 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 462 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,018 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection417: begin an election at term 460 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,814 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:54,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,035 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection417: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t460, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t460] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t460, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:54,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:54,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 460 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:54,279 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:54,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection418: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t471, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t471] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t471, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:54,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 471 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,091 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 469 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:54,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:54,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:54,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,307 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 463 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:54,091 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:54,120 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:85ms, electionTimeout:85ms datanode_2 | 2019-10-15 03:15:54,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:54,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,307 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:54,120 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 460 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:54,322 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 464 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:54,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,134 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection418: begin an election at term 461 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,157 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection418: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t461, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t461] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t461, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,322 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:54,157 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 461 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:54,416 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:54,157 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:20,869 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 672 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:54,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:20,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,416 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 471 for changeToCandidate datanode_1 | 2019-10-15 03:15:54,228 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 470 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:54,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:54,228 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,430 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection419: begin an election at term 472 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,869 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:54,228 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection419: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t472, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t472] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t472, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:20,869 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,228 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:20,968 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:54,446 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 472 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,272 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:54,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:54,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,446 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,272 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 461 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:54,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:54,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 465 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,276 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 471 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:54,276 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:54,276 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,485 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,276 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:54,485 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection419: begin an election at term 462 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:54,291 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection419: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t462, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t462] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t462, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:54,485 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:54,291 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 462 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:54,543 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:54,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:54,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:54,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 472 for changeToCandidate datanode_1 | 2019-10-15 03:15:54,292 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:54,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:54,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:54,556 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection420: begin an election at term 473 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,292 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 462 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:54,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:54,574 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection420: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t473, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t473] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t473, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,305 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection420: begin an election at term 463 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:54,574 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 473 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,308 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection420: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t463, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:21,018 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 673 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:54,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:21,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:21,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:54,639 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 466 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,321 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection420: begin an election at term 464 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection420: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t464, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t464] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t464, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:21,018 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,337 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 464 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:54,639 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:21,137 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:54,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:54,759 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:54,759 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:54,759 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 473 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:54,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:54,432 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 472 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:54,785 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection421: begin an election at term 474 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:54,814 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection421: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t474, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t474] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t474, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 474 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,432 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:54,469 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_2 | 2019-10-15 03:15:54,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:54,469 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,843 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:54,469 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 464 for changeToCandidate datanode_2 | 2019-10-15 03:15:54,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:54,469 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:54,844 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 474 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:54,483 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection421: begin an election at term 465 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:54,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:54,499 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection421: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t465, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t465] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t465, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:54,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 465 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:54,847 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection422: begin an election at term 475 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:54,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 467 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,500 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 473 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,137 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 674 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:54,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:21,137 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,854 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection422: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t475, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t475] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t475, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,558 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:21,138 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:54,854 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 475 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,614 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_3 | 2019-10-15 03:16:21,138 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,854 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:54,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,187 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 675 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 465 for changeToCandidate datanode_3 | 2019-10-15 03:16:21,187 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:54,854 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:21,187 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:54,897 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 468 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection422: begin an election at term 466 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:21,187 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,653 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection422: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t466, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t466] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t466, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:21,235 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:54,898 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,653 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 466 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:54,898 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:54,910 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_1 | 2019-10-15 03:15:54,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:54,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,789 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 474 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:54,911 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 475 for changeToCandidate datanode_1 | 2019-10-15 03:15:54,789 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:54,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:54,789 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:54,917 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection423: begin an election at term 476 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:54,789 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:54,918 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 469 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:54,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:54,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:54,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 466 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:54,918 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:54,953 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection423: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t476, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t476] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t476, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:54,847 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection423: begin an election at term 467 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:54,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 476 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:54,849 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 475 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:54,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:54,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,123 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 470 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:54,849 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,123 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,853 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection423: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t467, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t467] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t467, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,123 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:21,371 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 676 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:55,124 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 467 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:21,371 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,137 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_1 | 2019-10-15 03:15:54,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:21,371 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:54,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:21,371 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:55,137 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 476 for changeToCandidate datanode_1 | 2019-10-15 03:15:54,891 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_3 | 2019-10-15 03:16:21,455 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:54,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:54,891 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 467 for changeToCandidate datanode_1 | 2019-10-15 03:15:54,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:55,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:54,896 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection424: begin an election at term 468 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,141 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection424: begin an election at term 477 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,142 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 471 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:54,908 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection424: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t468, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:54,917 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection424: begin an election at term 469 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:54,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 476 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:55,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:55,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:54,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:55,142 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:54,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:55,156 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection424: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t477, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t477] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t477, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:54,918 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,156 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 477 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:54,953 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection424: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t469, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t469] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t469, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,156 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:54,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 469 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:55,156 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:54,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:55,218 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 472 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:54,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:55,097 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:55,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:55,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:55,218 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,097 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 469 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:55,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:15:55,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:55,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,121 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection425: begin an election at term 470 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:21,484 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 677 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:55,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection425: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t470, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:21,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:55,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection425: begin an election at term 471 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 477 for changeToCandidate datanode_3 | 2019-10-15 03:16:21,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,143 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 477 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:55,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:55,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,287 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection425: begin an election at term 478 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,143 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,303 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection425: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t478, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t478] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t478, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,143 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection425: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t471, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t471] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t471, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 478 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:21,615 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:55,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:55,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:55,358 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 473 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:55,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 471 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:55,358 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:55,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:55,358 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:55,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,358 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:55,203 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | 2019-10-15 03:15:55,467 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:55,203 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:55,203 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 471 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:55,203 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:55,467 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 478 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:55,217 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection426: begin an election at term 472 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:55,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection426: begin an election at term 479 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:55,233 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection426: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t472, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t472] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t472, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,498 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection426: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t479, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t479] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t479, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:55,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 472 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:55,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 479 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:55,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:55,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:55,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:55,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 478 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:55,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:21,618 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 678 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:55,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,522 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 474 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:21,618 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,522 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,618 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:55,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,522 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:21,618 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,342 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:15:55,522 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,737 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:55,575 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 475 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:55,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 472 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:55,575 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:55,575 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:55,356 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection427: begin an election at term 473 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,575 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,363 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection427: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t473, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t473] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t473, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,672 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_1 | 2019-10-15 03:15:55,363 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 473 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:55,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:55,672 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 479 for changeToCandidate datanode_1 | 2019-10-15 03:15:55,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:55,673 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:55,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 479 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:55,686 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection427: begin an election at term 480 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:55,716 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection427: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t480, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t480] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t480, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:55,717 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 480 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:55,483 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:55,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:55,495 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:55,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:55,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:55,495 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 473 for changeToCandidate datanode_2 | 2019-10-15 03:15:55,782 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 476 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:55,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:55,520 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection428: begin an election at term 474 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:55,536 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection428: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t474, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t474] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t474, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:55,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:55,537 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 474 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:55,782 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:55,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:55,879 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_3 | 2019-10-15 03:16:21,770 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 679 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:55,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,548 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_2 | 2019-10-15 03:15:55,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 480 for changeToCandidate datanode_3 | 2019-10-15 03:16:21,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:55,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:55,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:21,770 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,893 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection428: begin an election at term 481 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,549 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 474 for changeToCandidate datanode_3 | 2019-10-15 03:16:21,915 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:55,909 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection428: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t481, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t481] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t481, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,549 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:55,909 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 481 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:55,573 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection429: begin an election at term 475 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:55,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:55,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection429: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t475, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t475] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t475, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:55,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:55,601 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 475 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:55,925 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 477 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:55,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:55,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:55,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:55,688 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 480 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:55,925 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:55,688 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,963 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:55,688 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:55,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:55,688 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:55,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 481 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:55,730 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:15:55,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:55,730 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:55,979 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection429: begin an election at term 482 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,730 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 475 for changeToCandidate datanode_2 | 2019-10-15 03:15:56,007 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection429: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t482, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t482] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t482, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:55,730 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:55,780 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection430: begin an election at term 476 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,007 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 482 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:55,798 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection430: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t476, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t476] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t476, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:56,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:55,798 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 476 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:21,940 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 680 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:56,053 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_1 | 2019-10-15 03:15:55,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:21,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:21,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:55,894 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 481 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,053 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 482 for changeToCandidate datanode_3 | 2019-10-15 03:16:21,940 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:55,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:21,983 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:56,078 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection430: begin an election at term 483 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,895 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,095 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection430: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t483, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t483] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t483, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,910 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | 2019-10-15 03:15:56,095 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 483 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:55,910 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:55,910 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 476 for changeToCandidate datanode_2 | 2019-10-15 03:15:56,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:55,910 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 478 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:56,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:56,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:55,923 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection431: begin an election at term 477 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:56,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:55,939 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection431: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t477, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t477] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t477, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:56,161 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_1 | 2019-10-15 03:15:55,939 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 477 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:56,161 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:55,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:56,161 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 483 for changeToCandidate datanode_1 | 2019-10-15 03:15:55,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:56,161 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:55,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 482 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:56,166 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection431: begin an election at term 484 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:56,173 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection431: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t484, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t484] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t484, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:55,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:56,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 484 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:55,981 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:56,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:56,080 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 483 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:22,084 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:56,080 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,253 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_1 | 2019-10-15 03:15:56,080 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,080 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:56,253 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 484 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:56,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:56,114 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:15:56,268 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection432: begin an election at term 485 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:56,114 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,270 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 479 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:56,114 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 477 for changeToCandidate datanode_2 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:56,114 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,270 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:56,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection432: begin an election at term 478 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,286 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection432: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t485, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t485] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t485, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:56,128 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection432: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t478, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t478] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t478, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,286 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 485 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:56,128 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 478 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:56,286 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:56,286 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,128 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:56,297 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_1 | 2019-10-15 03:15:56,128 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:56,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,167 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 484 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:56,297 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 485 for changeToCandidate datanode_1 | 2019-10-15 03:15:56,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:56,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:56,168 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:22,110 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 681 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:56,323 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection433: begin an election at term 486 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,168 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,340 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection433: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t486, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t486] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t486, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,264 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:135ms, electionTimeout:135ms datanode_2 | 2019-10-15 03:15:56,340 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 486 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:56,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:22,110 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,264 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 478 for changeToCandidate datanode_3 | 2019-10-15 03:16:22,174 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 682 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:56,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:22,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,406 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_1 | 2019-10-15 03:15:56,268 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection433: begin an election at term 479 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:56,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 485 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:22,174 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:56,407 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 486 for changeToCandidate datanode_1 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:22,228 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 683 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:56,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:56,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:22,228 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:22,228 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:56,286 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection433: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t479, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t479] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t479, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,420 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection434: begin an election at term 487 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,228 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,287 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 479 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:56,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection434: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t487, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,299 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:56,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection434: begin an election at term 488 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:56,325 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 486 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:56,456 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection434: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t488, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t488] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t488, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:56,456 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 488 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:56,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:56,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:56,325 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:56,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 487 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:56,470 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 480 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:56,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:56,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:56,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:56,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:56,441 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 488 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:56,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,557 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:56,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:56,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,441 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:56,557 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 488 for changeToCandidate datanode_1 | 2019-10-15 03:15:56,454 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | 2019-10-15 03:15:56,557 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:56,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,582 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection435: begin an election at term 489 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:56,454 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 479 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:56,586 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 481 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:56,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,468 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection434: begin an election at term 480 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,436 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:56,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,487 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection434: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t480, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t480] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t480, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,586 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,487 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 480 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:56,600 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection435: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t489, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t489] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t489, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:56,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 489 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:56,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:56,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:56,578 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:91ms, electionTimeout:91ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:56,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,578 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:56,748 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_1 | 2019-10-15 03:15:56,579 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 480 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:56,579 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,748 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:56,584 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection435: begin an election at term 481 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,749 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 489 for changeToCandidate datanode_1 | 2019-10-15 03:15:56,584 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 489 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:56,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:56,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:56,763 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection436: begin an election at term 490 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:56,766 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 482 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:56,584 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:56,600 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection435: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t481, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t481] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t481, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,600 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 481 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:56,766 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,600 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:56,783 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection436: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t490, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t490] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t490, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,600 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:56,783 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 490 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:56,759 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_3 | 2019-10-15 03:16:22,464 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 684 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:56,783 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:22,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,783 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:22,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:56,759 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 481 for changeToCandidate datanode_2 | 2019-10-15 03:15:56,786 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_3 | 2019-10-15 03:16:22,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:56,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:22,514 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:56,763 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection436: begin an election at term 482 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,786 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 490 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:56,766 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 490 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,827 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection437: begin an election at term 491 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,852 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection437: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t491, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t491] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t491, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:56,766 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:56,852 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 491 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:56,766 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:56,852 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:56,783 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection436: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t482, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t482] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t482, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:56,852 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,783 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 482 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:56,911 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:56,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:56,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:56,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:56,911 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 491 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:56,828 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 491 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:56,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,925 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection438: begin an election at term 492 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:56,829 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:56,953 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection438: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t492, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t492] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t492, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:56,829 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:56,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 492 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:56,927 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 492 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:56,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:56,927 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:56,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,927 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:22,693 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 685 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:56,957 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 483 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:56,927 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:22,693 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:56,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,949 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_3 | 2019-10-15 03:16:22,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:56,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,949 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:22,694 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:56,957 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:56,950 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 482 for changeToCandidate datanode_3 | 2019-10-15 03:16:22,696 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:56,983 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_1 | 2019-10-15 03:15:56,950 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:56,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:56,955 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection437: begin an election at term 483 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,983 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 492 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:56,983 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection437: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t483, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t483] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t483, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:56,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:56,983 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 483 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:56,997 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection439: begin an election at term 493 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:57,015 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection439: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t493, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t493] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t493, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:56,984 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,016 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 493 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:56,999 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 493 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:57,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:56,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:57,016 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:56,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:57,049 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_1 | 2019-10-15 03:15:56,999 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:57,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,061 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:57,050 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 493 for changeToCandidate datanode_1 | 2019-10-15 03:15:57,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:57,051 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:57,061 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 483 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:57,075 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection440: begin an election at term 494 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,062 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:57,078 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 484 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,076 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection438: begin an election at term 484 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:57,078 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,078 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 494 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:57,102 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection440: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t494, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,104 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection440: begin an election at term 495 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,759 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 686 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,121 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection440: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t495, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t495] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t495, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:22,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,078 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 495 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:22,759 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:57,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:22,759 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,103 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection438: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t484, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t484] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t484, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:22,909 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:57,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 484 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:57,266 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 485 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:57,266 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:57,266 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,106 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 495 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:57,266 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:57,317 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_1 | 2019-10-15 03:15:57,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:57,317 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,106 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:57,317 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 495 for changeToCandidate datanode_1 | 2019-10-15 03:15:57,248 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:57,317 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:57,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:57,331 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 486 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,248 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 484 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:57,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:57,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,264 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection439: begin an election at term 485 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,331 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:57,293 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection439: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t485, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t485] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t485, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,343 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection441: begin an election at term 496 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:57,293 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 485 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:57,371 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection441: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t496, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t496] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t496, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,293 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:57,371 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 496 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:57,293 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:57,305 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_2 | 2019-10-15 03:15:57,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:57,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,406 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_3 | 2019-10-15 03:16:22,959 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 687 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,305 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 485 for changeToCandidate datanode_2 | 2019-10-15 03:15:57,406 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:22,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,305 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:57,406 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 496 for changeToCandidate datanode_3 | 2019-10-15 03:16:22,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:57,330 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection440: begin an election at term 486 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,406 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:22,960 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,344 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection440: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t486, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t486] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t486, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,420 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection442: begin an election at term 497 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,050 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 688 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,344 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 486 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:57,429 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection442: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t497, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t497] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t497, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:23,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:57,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,429 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 497 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:23,050 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,345 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 496 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:57,429 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:23,077 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:57,345 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,429 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:57,345 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,449 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_1 | 2019-10-15 03:15:57,345 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,449 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:57,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 497 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:57,449 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 497 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:57,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,449 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:57,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection443: begin an election at term 498 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:57,461 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection443: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t498, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t498] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t498, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,455 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 498 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:57,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 498 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:57,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:57,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:57,455 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 487 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:57,535 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_2 | 2019-10-15 03:15:57,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:57,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:57,535 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 486 for changeToCandidate datanode_2 | 2019-10-15 03:15:57,563 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:57,591 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_1 | 2019-10-15 03:15:57,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:57,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection441: begin an election at term 487 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:57,591 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 498 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:57,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:57,621 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection444: begin an election at term 499 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,179 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 689 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,582 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection441: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t487, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t487] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t487, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,637 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection444: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t499, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t499] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t499, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,582 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 487 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:23,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:57,637 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 499 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:57,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:23,179 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:23,194 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:57,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,638 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:57,622 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 499 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:57,713 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:57,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:57,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 499 for changeToCandidate datanode_1 | 2019-10-15 03:15:57,623 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:57,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:57,623 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:57,720 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection445: begin an election at term 500 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,693 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:57,720 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 488 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:57,693 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:57,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,693 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 487 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:57,693 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:57,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,718 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection442: begin an election at term 488 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:57,720 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,721 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 500 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:57,735 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection445: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t500, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t500] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t500, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,721 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:57,735 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 500 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:57,721 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:57,721 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,735 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,735 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection442: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t488, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t488] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t488, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:57,735 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 488 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:57,739 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:57,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:57,739 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,327 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:57,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,739 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 500 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:57,765 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 501 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:57,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:57,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:57,739 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:57,765 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:57,763 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection446: begin an election at term 501 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:57,800 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_2 | 2019-10-15 03:15:57,791 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection446: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t501, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t501] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t501, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:57,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:57,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 501 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:57,800 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 488 for changeToCandidate datanode_2 | 2019-10-15 03:15:57,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:57,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:57,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:57,814 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection443: begin an election at term 489 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,816 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 489 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:57,841 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection443: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t489, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t489] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t489, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,816 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:57,816 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:57,816 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 489 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:57,822 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:57,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:57,822 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 501 for changeToCandidate datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 502 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:57,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,363 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:57,840 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection447: begin an election at term 502 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:57,855 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection447: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t502, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t502] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t502, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:57,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:57,855 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 502 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:57,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:57,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:57,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:57,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_1 | 2019-10-15 03:15:57,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 503 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:57,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:57,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:57,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 502 for changeToCandidate datanode_1 | 2019-10-15 03:15:57,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:57,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:57,918 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:57,916 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection448: begin an election at term 503 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,009 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:57,934 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection448: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t503, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t503] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t503, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:57,934 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 503 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:58,010 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 489 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:57,934 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:58,010 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:57,934 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,035 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection444: begin an election at term 490 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:58,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_1 | 2019-10-15 03:15:58,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 504 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:58,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:58,023 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 503 for changeToCandidate datanode_1 | 2019-10-15 03:15:58,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:58,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:23,366 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 690 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:58,036 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection449: begin an election at term 504 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,066 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection444: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t490, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t490] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t490, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,064 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 490 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 505 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:23,366 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,366 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 490 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:23,451 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:58,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:58,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,120 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:58,065 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,247 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:58,106 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection449: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t504, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:58,118 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection449: begin an election at term 505 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,247 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 490 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:58,142 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection449: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t505, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t505] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t505, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:58,260 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection445: begin an election at term 491 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,142 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 505 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:58,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection445: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t491, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t491] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t491, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:58,277 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 491 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:58,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:58,277 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:58,262 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 491 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:58,277 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,317 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 506 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:58,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:58,262 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:58,302 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_1 | 2019-10-15 03:15:58,317 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:58,302 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,366 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_2 | 2019-10-15 03:15:58,302 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 505 for changeToCandidate datanode_3 | 2019-10-15 03:16:23,452 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 691 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,302 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:58,366 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 491 for changeToCandidate datanode_2 | 2019-10-15 03:15:58,316 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection450: begin an election at term 506 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:23,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,347 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection450: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t506, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t506] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t506, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,384 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection446: begin an election at term 492 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:58,347 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 506 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:23,452 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:58,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:58,400 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection446: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t492, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t492] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t492, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:23,577 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 692 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:58,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:23,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:58,400 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 492 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:58,386 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 492 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:23,577 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:58,386 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:23,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 693 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:58,386 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,433 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_3 | 2019-10-15 03:16:23,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,386 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:58,448 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 493 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 492 for changeToCandidate datanode_3 | 2019-10-15 03:16:23,677 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:58,448 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:23,677 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:58,448 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,446 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection447: begin an election at term 493 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:58,448 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,463 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection447: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t493, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t493] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t493, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:58,483 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_1 | 2019-10-15 03:15:58,463 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 493 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:58,483 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:58,483 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 506 for changeToCandidate datanode_1 | 2019-10-15 03:15:58,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:58,483 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:58,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 507 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:58,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection451: begin an election at term 507 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:58,523 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection451: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t507, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t507] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t507, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,524 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 507 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:58,499 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:58,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:58,603 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_2 | 2019-10-15 03:15:58,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,603 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:58,618 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 494 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,603 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 493 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:58,618 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:58,618 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:58,603 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:58,618 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,616 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection448: begin an election at term 494 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:58,641 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection448: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t494, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t494] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t494, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,623 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:58,641 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 494 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:58,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,829 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:58,641 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:58,623 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 507 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:58,642 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:58,642 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 508 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:58,640 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection452: begin an election at term 508 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:58,657 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection452: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t508, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t508] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t508, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,657 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 508 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:58,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:58,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,642 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:58,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 495 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,642 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:58,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,642 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:58,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:58,699 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:58,715 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:58,720 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_1 | 2019-10-15 03:15:58,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 494 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:58,699 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:58,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:58,712 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection449: begin an election at term 495 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 508 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:58,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:58,736 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection453: begin an election at term 509 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,719 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection449: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t495, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t495] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t495, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,753 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection453: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t509, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t509] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t509, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:58,720 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 495 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:58,753 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 509 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:23,896 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 694 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:58,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:23,896 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:58,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:23,896 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:58,775 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 496 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:23,896 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:58,776 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 509 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:23,964 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 695 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:58,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,776 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:23,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,776 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:23,964 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:58,738 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:58,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 497 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:23,964 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:58,767 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_2 | 2019-10-15 03:15:58,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:23,967 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:58,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:58,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:58,904 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:58,767 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 495 for changeToCandidate datanode_2 | 2019-10-15 03:15:58,907 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:58,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:58,908 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:15:58,771 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection450: begin an election at term 496 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,908 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 509 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:58,781 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection450: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t496, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t496] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t496, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:58,908 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:58,781 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 496 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:58,921 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection454: begin an election at term 510 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:58,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:58,937 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection454: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t510, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t510] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t510, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:58,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:58,937 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 510 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:58,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:58,874 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_2 | 2019-10-15 03:15:58,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:58,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,056 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 498 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:58,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 496 for changeToCandidate datanode_2 | 2019-10-15 03:15:59,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:58,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:59,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:58,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection451: begin an election at term 497 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,056 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:58,922 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection451: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t497, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t497] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t497, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,093 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:58,922 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 497 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:59,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:58,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:24,127 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 696 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:59,093 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 510 for changeToCandidate datanode_1 | 2019-10-15 03:15:58,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:24,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:58,923 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 510 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:24,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:59,097 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection455: begin an election at term 511 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:24,127 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,112 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection455: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t511, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t511] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t511, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:58,923 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,144 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 697 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:59,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 511 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:58,923 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:24,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:59,039 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | 2019-10-15 03:16:24,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:59,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,039 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,283 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 499 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,039 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 497 for changeToCandidate datanode_2 | 2019-10-15 03:15:59,283 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,039 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:59,283 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,054 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection452: begin an election at term 498 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:24,144 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,283 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,161 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 698 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:59,298 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_1 | 2019-10-15 03:15:59,070 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection452: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t498, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t498] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t498, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:24,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,071 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 498 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:24,161 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:15:59,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 511 for changeToCandidate datanode_3 | 2019-10-15 03:16:24,161 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:59,071 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:24,197 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:59,324 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection456: begin an election at term 512 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:59,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 500 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:59,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:59,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:59,326 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:59,330 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection456: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t512, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,071 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,351 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection456: begin an election at term 513 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:59,366 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection456: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t513, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t513] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t513, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:59,366 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 513 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:59,366 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:59,366 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:59,367 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 501 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:59,367 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:59,367 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:59,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 511 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:59,367 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:59,434 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_3 | 2019-10-15 03:16:24,235 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:59,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,434 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 513 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:59,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:15:59,449 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection457: begin an election at term 514 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:59,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection457: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t514, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:59,485 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection457: begin an election at term 515 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:59,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,513 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection457: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t515, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t515] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t515, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:59,513 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 515 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:59,513 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:59,513 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:59,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,526 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:59,099 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:59,256 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_2 | 2019-10-15 03:15:59,527 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 515 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:59,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,527 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:59,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 498 for changeToCandidate datanode_2 | 2019-10-15 03:15:59,530 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: begin an election at term 516 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:59,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:15:59,534 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t516, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,281 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection453: begin an election at term 499 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:59,538 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: begin an election at term 517 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,298 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection453: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t499, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t499] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t499, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t517, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,298 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 499 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:15:59,547 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: begin an election at term 518 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,298 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:59,569 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection458: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t518, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t518] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t518, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,569 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 518 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:59,298 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:24,263 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:15:59,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:59,308 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | 2019-10-15 03:15:59,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:15:59,570 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 502 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,308 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 499 for changeToCandidate datanode_2 | 2019-10-15 03:15:59,570 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,570 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:15:59,570 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:59,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection454: begin an election at term 500 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:15:59,607 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_1 | 2019-10-15 03:15:59,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 512 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:59,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:59,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,607 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 518 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:59,326 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:59,345 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection454: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t500, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection459: begin an election at term 519 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:59,352 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 513 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:59,655 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection459: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t519, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t519] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t519, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:59,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,655 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 519 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:59,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:59,352 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:59,365 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection454: begin an election at term 501 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:59,397 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection454: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t501, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t501] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t501, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:15:59,657 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 503 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:59,397 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 501 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:15:59,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:59,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:15:59,657 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:15:59,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,657 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,383 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 699 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:15:59,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 504 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,451 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 514 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:59,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:24,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:24,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:59,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,688 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,383 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:59,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:15:59,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms datanode_3 | 2019-10-15 03:16:24,441 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:59,451 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:59,487 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 515 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:15:59,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 519 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:59,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:59,488 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:15:59,781 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:15:59,794 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection460: begin an election at term 520 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:15:59,813 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection460: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t520, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t520] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t520, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:15:59,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 520 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:15:59,532 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 516 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:15:59,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:59,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:15:59,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:15:59,921 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 505 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,532 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:15:59,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,539 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 517 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:15:59,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:15:59,921 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:59,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:15:59,996 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_1 | 2019-10-15 03:15:59,539 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:15:59,996 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,548 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 518 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:15:59,996 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 520 for changeToCandidate datanode_1 | 2019-10-15 03:15:59,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:15:59,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:15:59,996 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:59,548 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,583 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 700 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:00,011 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection461: begin an election at term 521 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,556 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_3 | 2019-10-15 03:16:24,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,025 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection461: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t521, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:24,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:15:59,556 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 501 for changeToCandidate datanode_3 | 2019-10-15 03:16:24,583 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:59,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:00,049 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection461: begin an election at term 522 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:24,623 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:15:59,568 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection455: begin an election at term 502 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,075 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection461: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t522, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t522] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t522, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:15:59,583 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection455: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t502, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t502] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t502, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,075 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 522 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:15:59,583 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 502 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:00,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:59,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:00,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,584 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:15:59,613 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 519 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:00,106 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 506 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:59,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,106 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:59,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,107 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:59,613 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,107 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:15:59,650 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:16:00,213 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:15:59,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,213 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:15:59,650 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 502 for changeToCandidate datanode_2 | 2019-10-15 03:16:00,213 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 522 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:15:59,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:00,213 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:59,656 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection456: begin an election at term 503 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection462: begin an election at term 523 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:15:59,656 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection456: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t503, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,218 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 507 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,686 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection456: begin an election at term 504 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:00,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,713 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection456: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t504, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t504] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t504, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:00,218 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 504 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:24,713 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 701 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:00,218 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:15:59,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:24,713 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection462: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t523, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t523] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t523, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:15:59,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,261 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 523 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:24,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:16:24,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:15:59,796 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 520 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:24,805 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:00,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:15:59,796 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:00,344 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_1 | 2019-10-15 03:15:59,796 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:00,344 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,796 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:15:59,907 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:00,344 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 523 for changeToCandidate datanode_1 | 2019-10-15 03:15:59,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:00,344 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:15:59,907 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 504 for changeToCandidate datanode_2 | 2019-10-15 03:16:00,357 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection463: begin an election at term 524 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:15:59,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:00,375 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection463: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t524, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t524] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t524, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:15:59,919 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection457: begin an election at term 505 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,375 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 524 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:15:59,943 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection457: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t505, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t505] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t505, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:00,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:00,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 508 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:15:59,943 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 505 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:00,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:15:59,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:15:59,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:00,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 521 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:00,376 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:00,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:00,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,514 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_1 | 2019-10-15 03:16:00,013 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:24,883 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:00,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,051 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 522 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:00,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,514 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 524 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:00,051 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:00,051 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,529 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection464: begin an election at term 525 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:00,555 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection464: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t525, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t525] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t525, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:00,090 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_2 | 2019-10-15 03:16:00,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 525 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:00,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:00,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 505 for changeToCandidate datanode_2 | 2019-10-15 03:16:00,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:00,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:00,582 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 509 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,105 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection458: begin an election at term 506 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:00,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,121 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection458: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t506, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t506] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t506, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:00,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,121 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 506 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:00,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:00,647 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 510 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:00,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,202 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:00,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:00,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,202 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 506 for changeToCandidate datanode_3 | 2019-10-15 03:16:24,884 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 702 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:00,706 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_3 | 2019-10-15 03:16:24,884 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:24,884 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:00,706 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 525 for changeToCandidate datanode_1 | 2019-10-15 03:16:00,216 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection459: begin an election at term 507 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:24,884 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:00,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 523 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:25,010 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 703 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,720 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection465: begin an election at term 526 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:25,010 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:00,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,738 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection465: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t526, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t526] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t526, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:00,219 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,738 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 526 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:00,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection459: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t507, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t507] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t507, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:00,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:00,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,245 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 507 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:00,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 511 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:00,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:00,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,358 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:00,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:00,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 512 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,358 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 507 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:00,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:00,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,359 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 524 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:00,850 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:00,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,936 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_1 | 2019-10-15 03:16:00,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:00,360 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,937 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 526 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:00,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:00,374 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection460: begin an election at term 508 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:00,950 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection466: begin an election at term 527 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:00,381 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection460: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t508, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t508] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t508, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:00,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 508 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:25,010 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:25,011 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:00,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,968 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection466: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t527, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t527] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t527, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:25,011 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,531 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 525 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:25,234 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 704 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,968 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 527 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:00,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:00,531 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:00,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:25,234 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,567 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_2 | 2019-10-15 03:16:00,986 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 513 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:25,234 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:00,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:00,986 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:00,567 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 508 for changeToCandidate datanode_2 | 2019-10-15 03:16:00,986 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:00,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:00,986 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:00,580 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection461: begin an election at term 509 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:01,113 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_1 | 2019-10-15 03:16:00,608 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection461: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t509, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t509] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t509, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:01,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,608 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 509 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:01,113 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 527 for changeToCandidate datanode_1 | 2019-10-15 03:16:00,608 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:01,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:00,608 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:00,631 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:00,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,127 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection467: begin an election at term 528 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:01,145 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection467: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t528, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t528] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t528, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:00,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 509 for changeToCandidate datanode_2 | 2019-10-15 03:16:01,146 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 528 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:00,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:01,146 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:00,645 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection462: begin an election at term 510 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:01,146 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:01,228 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 514 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,661 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection462: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t510, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t510] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t510, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:01,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,661 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 510 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:25,400 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 705 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:01,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:01,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:01,234 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_1 | 2019-10-15 03:16:00,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:25,400 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:00,722 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 526 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:01,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 528 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:00,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:00,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,254 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection468: begin an election at term 529 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:01,274 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection468: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t529, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t529] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t529, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:00,722 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:01,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 529 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:00,764 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:01,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:00,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:00,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 510 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:00,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:01,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:00,811 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection463: begin an election at term 511 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:00,827 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection463: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t511, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t511] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t511, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:00,828 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 511 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:01,354 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 529 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:00,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:01,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:00,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:00,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_2 | 2019-10-15 03:16:01,368 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection469: begin an election at term 530 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:01,396 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection469: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t530, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t530] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t530, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:00,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,396 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 530 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:00,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 511 for changeToCandidate datanode_3 | 2019-10-15 03:16:25,400 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:01,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:00,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:00,848 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection464: begin an election at term 512 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,864 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection464: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t512, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t512] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t512, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,420 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_3 | 2019-10-15 03:16:25,460 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:00,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 512 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:00,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:01,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:00,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,420 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 530 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:01,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:00,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 527 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:01,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection470: begin an election at term 531 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:00,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,437 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 515 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:00,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:01,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:00,952 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:01,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:00,979 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:01,437 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:00,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:01,463 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection470: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t531, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t531] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t531, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:00,979 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 512 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:01,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 531 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:00,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:01,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:00,984 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection465: begin an election at term 513 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:01,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection465: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t513, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t513] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t513, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:01,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 516 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:01,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:01,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 513 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:01,466 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:01,620 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_1 | 2019-10-15 03:16:01,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:01,620 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,129 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 528 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:25,508 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 706 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:01,620 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 531 for changeToCandidate datanode_3 | 2019-10-15 03:16:25,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,620 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:01,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:25,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:01,626 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection471: begin an election at term 532 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:25,508 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,626 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 517 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:01,129 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:25,543 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 707 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:01,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,210 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | 2019-10-15 03:16:01,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:25,543 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,626 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 513 for changeToCandidate datanode_3 | 2019-10-15 03:16:25,543 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:01,653 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection471: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t532, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t532] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t532, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:01,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:25,543 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:01,653 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 532 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:01,225 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection466: begin an election at term 514 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:25,675 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:01,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:01,256 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection466: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t514, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t514] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t514, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 514 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:01,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:01,257 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:01,655 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 518 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:01,257 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 529 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:01,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,257 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,257 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:01,655 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,257 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 519 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:01,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 530 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:01,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:01,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:01,670 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,370 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:01,431 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | 2019-10-15 03:16:01,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:01,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:01,431 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 514 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:01,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:01,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 532 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:01,435 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection467: begin an election at term 515 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:01,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 531 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:01,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:01,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,843 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection472: begin an election at term 533 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:01,437 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,845 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 520 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:01,437 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:25,754 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 708 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:01,458 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection467: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t515, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:25,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,464 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection467: begin an election at term 516 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,845 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,480 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection467: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t516, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t516] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t516, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:01,859 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection472: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t533, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t533] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t533, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:25,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:01,480 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 516 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:01,859 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 533 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:25,754 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:01,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:25,780 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:01,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,981 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:01,599 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | 2019-10-15 03:16:01,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,981 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 533 for changeToCandidate datanode_2 | 2019-10-15 03:16:01,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:01,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 516 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:01,995 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection473: begin an election at term 534 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:01,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:01,997 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 521 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:01,624 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: begin an election at term 517 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:01,628 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 532 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:01,628 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:01,628 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:01,628 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:01,997 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:01,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t517, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,024 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection473: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t534, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t534] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t534, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:01,653 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: begin an election at term 518 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,024 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 534 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:01,665 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t518, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:01,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: begin an election at term 519 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:01,677 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection468: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t519, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t519] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t519, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,050 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_1 | 2019-10-15 03:16:01,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 519 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:02,051 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:02,051 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 534 for changeToCandidate datanode_1 | 2019-10-15 03:16:01,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:02,051 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:01,826 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:02,057 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection474: begin an election at term 535 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:01,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:02,066 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection474: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t535, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t535] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t535, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:01,827 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 519 for changeToCandidate datanode_3 | 2019-10-15 03:16:25,872 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:02,066 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 535 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:01,827 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:02,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:01,843 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection469: begin an election at term 520 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:02,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,845 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 533 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:02,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_1 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:02,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,845 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:02,154 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 535 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:01,845 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,154 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:01,859 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection469: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t520, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t520] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t520, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:01,859 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 520 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:02,168 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection475: begin an election at term 536 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:01,859 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:02,201 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection475: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t536, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t536] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t536, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:01,859 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,201 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 536 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:01,986 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | 2019-10-15 03:16:02,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:01,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:01,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 520 for changeToCandidate datanode_2 | 2019-10-15 03:16:02,202 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 522 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:01,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:02,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:01,995 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection470: begin an election at term 521 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:02,203 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:25,918 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:01,997 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 534 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:02,230 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 523 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:02,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:02,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:01,997 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:02,230 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:01,997 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:02,246 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 524 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection470: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t521, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t521] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t521, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:02,246 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 521 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:02,246 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:02,246 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:02,279 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 525 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:02,059 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 535 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:02,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:02,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,279 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:02,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:02,059 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:02,170 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 536 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:02,290 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 536 for changeToCandidate datanode_1 | 2019-10-15 03:16:02,170 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:02,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:02,171 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,294 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection476: begin an election at term 537 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,171 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:02,301 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection476: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t537, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t537] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t537, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,178 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_2 | 2019-10-15 03:16:02,301 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 537 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:02,178 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,301 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:25,918 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 709 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,178 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 521 for changeToCandidate datanode_2 | 2019-10-15 03:16:02,301 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:25,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,178 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:25,919 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:02,200 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: begin an election at term 522 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,436 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_3 | 2019-10-15 03:16:25,919 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,205 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t522, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,436 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:25,978 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:02,228 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: begin an election at term 523 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,436 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 537 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:02,230 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t523, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,436 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:02,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: begin an election at term 524 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,450 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection477: begin an election at term 538 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:02,257 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection471: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t524, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t524] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t524, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,469 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection477: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t538, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:02,257 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 524 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:02,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 526 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,257 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:02,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,257 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:02,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:02,272 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:02,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:02,478 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,272 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 524 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:02,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:02,479 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection477: begin an election at term 539 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:02,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection472: begin an election at term 525 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:02,493 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection477: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t539, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t539] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t539, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,286 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection472: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t525, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t525] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t525, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:02,493 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 539 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:02,286 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 525 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:02,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:02,286 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:02,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:02,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,615 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 527 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:02,295 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 537 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:02,615 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:02,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,615 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,014 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 710 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,615 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:26,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,295 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,660 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_3 | 2019-10-15 03:16:26,014 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:02,451 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_3 | 2019-10-15 03:16:26,015 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,030 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 711 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,451 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 525 for changeToCandidate datanode_3 | 2019-10-15 03:16:26,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:26,030 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:02,660 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 539 for changeToCandidate datanode_1 | 2019-10-15 03:16:02,452 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 538 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:26,030 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:02,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,213 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 712 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:02,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection478: begin an election at term 540 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,677 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 528 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,452 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:26,213 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:02,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection473: begin an election at term 526 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:02,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:02,480 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 539 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:02,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,677 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:02,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,703 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection478: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t540, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t540] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t540, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:02,480 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 540 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:02,704 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:02,493 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection473: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t526, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t526] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t526, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,704 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:02,493 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 526 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:02,886 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:02,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:02,886 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:02,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:02,886 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 540 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:02,599 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | 2019-10-15 03:16:02,886 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:02,887 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 529 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:02,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:02,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 526 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:02,887 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:02,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:02,887 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,613 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection474: begin an election at term 527 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:02,901 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection479: begin an election at term 541 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection474: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t527, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:02,946 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection479: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t541, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t541] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t541, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,675 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection474: begin an election at term 528 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:02,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 541 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:02,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 540 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:26,213 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:02,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:02,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:02,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:02,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,106 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_1 | 2019-10-15 03:16:02,677 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:26,329 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:02,703 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection474: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t528, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t528] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t528, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:03,106 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:03,106 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 541 for changeToCandidate datanode_1 | 2019-10-15 03:16:02,703 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 528 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:03,106 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:03,116 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 530 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:02,703 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:03,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:02,703 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:02,871 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | 2019-10-15 03:16:03,116 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:02,871 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,117 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection480: begin an election at term 542 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:02,871 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 528 for changeToCandidate datanode_2 | 2019-10-15 03:16:03,131 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection480: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t542, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t542] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t542, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:03,131 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 542 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:03,131 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:03,132 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:02,871 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:03,191 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:03,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:02,886 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection475: begin an election at term 529 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:03,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 542 for changeToCandidate datanode_1 | 2019-10-15 03:16:02,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection475: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t529, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t529] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t529, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:03,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 529 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:03,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection481: begin an election at term 543 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:03,235 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection481: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t543, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t543] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t543, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:26,357 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 713 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:03,236 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 543 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 541 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:26,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:26,357 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,357 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:02,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,300 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 531 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:26,413 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:02,903 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:03,088 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_2 | 2019-10-15 03:16:03,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:03,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,300 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:03,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 529 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:03,433 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_1 | 2019-10-15 03:16:03,089 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:03,433 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,114 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection476: begin an election at term 530 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:03,433 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 543 for changeToCandidate datanode_1 | 2019-10-15 03:16:03,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 542 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:03,433 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:03,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection482: begin an election at term 544 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:03,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,475 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection482: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t544, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t544] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t544, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:03,119 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,475 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 544 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:03,131 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection476: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t530, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t530] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t530, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:03,475 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:03,131 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 530 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:03,475 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:03,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:03,512 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 532 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:03,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,512 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:03,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 543 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:03,513 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:03,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:03,220 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,513 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:03,284 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:16:03,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 533 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:03,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:03,284 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 530 for changeToCandidate datanode_2 | 2019-10-15 03:16:03,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 714 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:03,563 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:03,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:26,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,570 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:94ms, electionTimeout:94ms datanode_1 | 2019-10-15 03:16:03,298 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection477: begin an election at term 531 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:26,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:03,570 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,442 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:03,315 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection477: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t531, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t531] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t531, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:03,570 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 544 for changeToCandidate datanode_3 | 2019-10-15 03:16:26,469 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 715 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:03,315 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 531 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:03,570 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:26,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,469 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:03,584 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection483: begin an election at term 545 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:03,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:26,469 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,599 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection483: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t545, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t545] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t545, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:03,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:26,588 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 716 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:03,599 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 545 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:03,456 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 544 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:03,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:03,456 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:03,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,588 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,456 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,667 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_3 | 2019-10-15 03:16:26,588 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:03,456 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,588 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:03,667 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 545 for changeToCandidate datanode_1 | 2019-10-15 03:16:03,496 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_3 | 2019-10-15 03:16:26,637 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:03,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:03,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:03,698 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection484: begin an election at term 546 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:03,496 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 531 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:03,700 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 534 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:03,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,511 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection478: begin an election at term 532 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:03,539 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection478: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t532, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t532] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t532, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:03,700 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:03,539 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 532 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:03,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:03,714 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection484: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t546, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t546] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t546, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:03,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 546 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:03,548 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | 2019-10-15 03:16:03,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:03,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:03,715 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:03,548 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 532 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:03,747 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_1 | 2019-10-15 03:16:03,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:03,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection479: begin an election at term 533 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:03,747 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 546 for changeToCandidate datanode_1 | 2019-10-15 03:16:03,585 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection479: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t533, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t533] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t533, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:03,585 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 533 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:03,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:26,716 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 717 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:03,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:03,586 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 545 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:16:03,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,586 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:26,716 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:03,660 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:03,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:03,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:03,772 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection485: begin an election at term 547 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:03,660 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 533 for changeToCandidate datanode_2 | 2019-10-15 03:16:03,788 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection485: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t547, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t547] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t547, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:03,660 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:03,788 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 547 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:03,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:03,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection480: begin an election at term 534 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:03,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:03,699 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 546 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:03,925 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 535 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:03,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:03,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,700 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:03,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:03,714 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection480: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t534, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t534] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t534, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:03,925 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:03,714 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 534 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:03,963 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:03,714 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:03,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:03,714 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 547 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:03,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 547 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:03,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:03,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:03,977 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection486: begin an election at term 548 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:26,716 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:03,774 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection486: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t548, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t548] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t548, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:03,897 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | 2019-10-15 03:16:04,005 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 548 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:03,897 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:26,865 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:03,897 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 534 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:04,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:03,897 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:04,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:03,923 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection481: begin an election at term 535 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:04,007 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 536 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:03,948 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection481: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t535, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t535] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t535, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:04,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:03,948 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 535 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:04,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:03,948 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,007 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:03,948 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,010 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:04,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:04,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 548 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:03,978 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 548 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:04,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:03,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,033 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection487: begin an election at term 549 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:03,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,036 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 537 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:03,979 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,036 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:03,997 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_2 | 2019-10-15 03:16:04,036 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:03,997 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,036 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:03,997 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 535 for changeToCandidate datanode_2 | 2019-10-15 03:16:04,049 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection487: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t549, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t549] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t549, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:03,998 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:04,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 549 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:26,899 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 718 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:04,005 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: begin an election at term 536 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:26,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t536, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:26,899 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:04,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 549 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:04,075 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 538 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:26,900 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:04,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:27,003 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:04,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:04,034 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,075 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:04,035 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: begin an election at term 537 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,182 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:04,059 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t537, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,182 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:04,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: begin an election at term 538 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,182 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 549 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:04,182 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:04,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection482: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t538, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t538] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t538, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:04,196 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection488: begin an election at term 550 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:04,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 538 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:04,223 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection488: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t550, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t550] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t550, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:04,223 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 550 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:04,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:04,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:04,198 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 550 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:04,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,198 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:04,233 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_1 | 2019-10-15 03:16:04,198 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:04,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,198 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:04,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 550 for changeToCandidate datanode_1 | 2019-10-15 03:16:04,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 551 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:04,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:04,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:04,264 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection489: begin an election at term 551 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:27,029 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:04,281 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection489: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t551, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t551] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t551, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,266 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 551 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:04,282 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | 2019-10-15 03:16:04,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:04,282 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:04,282 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 538 for changeToCandidate datanode_2 | 2019-10-15 03:16:04,297 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 539 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:04,283 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:04,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection483: begin an election at term 539 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:04,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,323 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection483: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t539, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t539] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t539, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:04,298 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:04,323 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 539 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:04,354 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 540 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:04,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:04,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:04,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:04,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,338 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:04,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:16:04,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:04,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 539 for changeToCandidate datanode_2 | 2019-10-15 03:16:04,388 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 551 for changeToCandidate datanode_1 | 2019-10-15 03:16:04,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:04,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:04,351 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection484: begin an election at term 540 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:04,404 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection490: begin an election at term 552 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:04,380 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection484: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t540, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t540] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t540, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,405 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 541 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:27,032 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 719 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:04,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 540 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:27,032 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,406 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:27,032 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:04,393 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | 2019-10-15 03:16:04,421 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection490: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t552, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t552] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t552, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,032 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:04,393 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,421 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 552 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:27,048 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:04,393 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 540 for changeToCandidate datanode_2 | 2019-10-15 03:16:04,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:04,393 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:04,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:04,404 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection485: begin an election at term 541 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,571 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:04,406 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 552 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,571 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,571 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 552 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:04,406 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,571 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:04,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection485: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t541, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t541] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t541, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,585 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection491: begin an election at term 553 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:04,587 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 542 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:04,420 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 541 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:04,571 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_2 | 2019-10-15 03:16:04,587 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:04,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,601 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection491: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t553, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t553] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t553, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:04,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 541 for changeToCandidate datanode_2 | 2019-10-15 03:16:04,601 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 553 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:04,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:04,585 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection486: begin an election at term 542 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:04,587 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 553 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:04,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:04,637 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 543 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:04,587 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:04,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,587 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:04,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection486: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t542, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t542] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t542, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,637 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:27,192 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 720 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:04,601 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 542 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:04,709 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:16:27,192 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:27,192 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:04,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,709 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 553 for changeToCandidate datanode_1 | 2019-10-15 03:16:04,610 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_3 | 2019-10-15 03:16:27,192 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:04,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:04,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:27,196 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:04,735 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection492: begin an election at term 554 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,610 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 542 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:04,762 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection492: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t554, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t554] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t554, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:04,762 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 554 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:04,635 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection487: begin an election at term 543 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:04,762 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:04,664 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection487: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t543, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t543] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t543, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:04,664 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 543 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:04,762 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:04,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 544 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:04,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:04,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:04,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:04,737 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 554 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:04,811 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:04,843 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_1 | 2019-10-15 03:16:04,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:04,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:04,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:04,844 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 554 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:04,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:04,737 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:04,859 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection493: begin an election at term 555 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,794 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:04,878 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection493: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t555, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t555] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t555, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:04,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 555 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:04,794 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 543 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:04,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:04,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:04,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:04,809 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection488: begin an election at term 544 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,318 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:04,836 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection488: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t544, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t544] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t544, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:04,962 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:83ms, electionTimeout:83ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:04,837 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 544 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:04,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:04,962 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 555 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:04,837 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:04,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:04,987 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection494: begin an election at term 556 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:04,837 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:04,863 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 555 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:04,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,005 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection494: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t556, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t556] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t556, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:04,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:04,863 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 556 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:04,989 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 556 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:05,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:04,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:04,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:04,990 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:05,025 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_2 | 2019-10-15 03:16:05,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:05,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,042 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 545 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:05,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 544 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:05,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:27,398 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 721 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:05,042 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:05,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection489: begin an election at term 545 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 546 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:05,069 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection489: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t545, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t545] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t545, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:05,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,070 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 545 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:05,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:27,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:05,070 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:05,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:27,472 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:05,070 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,205 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:05,093 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_2 | 2019-10-15 03:16:05,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,206 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 556 for changeToCandidate datanode_1 | 2019-10-15 03:16:05,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 545 for changeToCandidate datanode_1 | 2019-10-15 03:16:05,094 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:05,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection490: begin an election at term 546 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,206 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:05,148 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection490: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t546, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t546] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t546, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,220 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection495: begin an election at term 557 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:05,148 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 546 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:05,262 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection495: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t557, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t557] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t557, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:05,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:05,262 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 557 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:05,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:05,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:05,221 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 557 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:05,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:05,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 547 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:05,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:05,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,222 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,237 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:05,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:05,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:05,237 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 546 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:05,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 548 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:05,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection491: begin an election at term 547 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection491: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t547, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t547] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t547, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 547 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:05,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:05,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:05,424 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:05,409 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:05,424 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_1 | 2019-10-15 03:16:05,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,409 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 547 for changeToCandidate datanode_3 | 2019-10-15 03:16:27,614 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 722 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 557 for changeToCandidate datanode_1 | 2019-10-15 03:16:05,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:27,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:05,422 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection492: begin an election at term 548 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:05,438 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection496: begin an election at term 558 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,614 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:05,439 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection492: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t548, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t548] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t548, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection496: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t558, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t558] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t558, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,644 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:05,439 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 548 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:05,454 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 558 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:05,439 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:05,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:05,439 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,454 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,440 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 558 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:05,472 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_1 | 2019-10-15 03:16:05,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:05,472 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:05,440 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:05,473 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 558 for changeToCandidate datanode_1 | 2019-10-15 03:16:05,440 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:05,479 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 559 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:05,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:05,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:05,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,478 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection497: begin an election at term 559 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:05,480 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,493 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection497: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t559, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t559] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t559, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:05,539 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 560 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:05,493 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 559 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:05,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:05,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:05,539 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,523 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:05,569 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:16:05,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:05,569 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,523 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 559 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:05,569 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 548 for changeToCandidate datanode_2 | 2019-10-15 03:16:05,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:27,768 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:05,537 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection498: begin an election at term 560 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:05,569 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:05,554 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection498: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t560, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t560] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t560, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:05,596 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection493: begin an election at term 549 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:05,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 560 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:05,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 561 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:05,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:05,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:05,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:05,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,573 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:05,598 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:05,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection493: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t549, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t549] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t549, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:05,574 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 560 for changeToCandidate datanode_1 | 2019-10-15 03:16:05,615 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 549 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:05,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:05,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:05,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:05,596 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection499: begin an election at term 561 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:05,598 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 549 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:05,800 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_2 | 2019-10-15 03:16:05,598 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:05,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:05,800 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 549 for changeToCandidate datanode_2 | 2019-10-15 03:16:05,599 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:05,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:05,615 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection499: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t561, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t561] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t561, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:05,615 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 561 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:06,109 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection494: begin an election at term 550 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:05,615 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:27,796 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 723 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:06,111 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 562 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:05,616 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:27,796 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:05,782 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_3 | 2019-10-15 03:16:27,796 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:27,796 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:05,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,111 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:27,863 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:05,782 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 561 for changeToCandidate datanode_1 | 2019-10-15 03:16:06,141 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection494: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t550, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t550] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t550, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:05,782 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:06,141 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 550 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:06,109 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection500: begin an election at term 562 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:06,111 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 550 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:06,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,195 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 563 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:06,111 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:06,111 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:06,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection500: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t562, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t562] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t562, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:06,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:06,195 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,141 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 562 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:06,272 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 564 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:06,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:06,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:06,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,176 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_1 | 2019-10-15 03:16:06,272 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,176 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:06,272 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,177 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 562 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:06,318 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_2 | 2019-10-15 03:16:06,177 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:06,193 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection501: begin an election at term 563 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:06,212 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection501: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t563, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t563] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t563, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,318 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 550 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:06,212 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 563 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:06,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:06,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:06,332 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection495: begin an election at term 551 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:27,890 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:06,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:06,355 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection495: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t551, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t551] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t551, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:06,253 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_1 | 2019-10-15 03:16:06,355 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 551 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:06,253 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,355 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:06,253 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 563 for changeToCandidate datanode_1 | 2019-10-15 03:16:06,355 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:06,254 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:06,405 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 565 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:06,270 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection502: begin an election at term 564 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:06,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection502: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t564, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t564] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t564, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:06,405 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 564 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:06,552 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | 2019-10-15 03:16:06,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:06,552 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:06,552 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 551 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:06,334 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 551 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:06,552 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:06,334 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,578 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection496: begin an election at term 552 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:06,606 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection496: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t552, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t552] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t552, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:06,334 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:06,334 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:06,606 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 552 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:06,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_1 | 2019-10-15 03:16:06,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:06,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:28,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 724 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:06,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 564 for changeToCandidate datanode_3 | 2019-10-15 03:16:28,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:28,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:06,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:06,403 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection503: begin an election at term 565 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:06,438 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection503: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t565, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t565] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t565, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,006 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,438 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 565 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:28,033 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 725 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:06,640 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 566 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:06,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:28,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:28,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:06,580 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 552 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:06,640 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:28,033 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:06,640 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:28,063 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:06,580 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,640 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:06,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 567 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:06,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,580 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:06,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,580 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:06,670 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,624 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:06,746 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | 2019-10-15 03:16:06,625 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:06,746 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,625 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 565 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:06,625 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:06,746 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 552 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:06,746 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:06,638 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection504: begin an election at term 566 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:06,772 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection497: begin an election at term 553 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:06,660 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection504: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t566, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:06,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection497: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t553, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t553] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t553, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:06,800 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 553 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:06,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:06,668 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection504: begin an election at term 567 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,689 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection504: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t567, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t567] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t567, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,843 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 568 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:06,689 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 567 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:06,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,689 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:06,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,689 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:06,843 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:06,773 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 553 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:06,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:06,942 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 569 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:06,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:28,190 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:06,942 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:06,942 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,827 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:06,827 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:06,942 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:06,827 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 567 for changeToCandidate datanode_1 | 2019-10-15 03:16:06,990 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:06,827 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:06,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:06,841 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection505: begin an election at term 568 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 553 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:06,859 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection505: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t568, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t568] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t568, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:06,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:06,859 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 568 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:07,004 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection498: begin an election at term 554 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:06,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:07,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection498: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t554, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t554] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t554, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:06,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 554 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:07,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:06,926 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:07,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:06,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:07,037 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | 2019-10-15 03:16:06,926 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 568 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:07,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:07,037 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 554 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:06,940 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection506: begin an election at term 569 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:06,956 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection506: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t569, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t569] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t569, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,050 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection499: begin an election at term 555 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,205 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 726 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:06,956 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 569 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:28,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:06,956 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:07,067 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection499: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t555, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t555] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t555, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:06,956 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:28,205 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,067 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 555 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:07,007 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 554 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:28,323 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:07,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:07,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,098 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:07,007 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,052 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 555 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:07,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 555 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:07,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:07,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection500: begin an election at term 556 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,053 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:07,109 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection500: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t556, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t556] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t556, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 556 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:07,109 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 556 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:07,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:07,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:07,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:07,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:07,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 570 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,139 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:07,139 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:07,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,139 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 569 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:07,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,139 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:07,155 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,153 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: begin an election at term 570 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:07,181 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 571 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,156 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t570, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:07,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,179 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: begin an election at term 571 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,400 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 727 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,181 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t571, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,400 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:07,181 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: begin an election at term 572 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:07,197 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 572 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,211 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection507: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t572, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t572] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t572, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:07,197 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,211 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 572 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:07,197 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:07,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,197 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:07,268 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 557 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,254 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:07,268 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:07,269 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,254 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 556 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,269 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:07,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,369 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:07,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection501: begin an election at term 557 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:07,283 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection501: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t557, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t557] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t557, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:07,284 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 557 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:07,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:07,369 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 572 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:07,284 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:07,380 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 573 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,378 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection508: begin an election at term 573 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:07,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:07,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:28,400 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,406 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection508: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t573, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t573] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t573, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,380 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:28,401 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:07,385 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_2 | 2019-10-15 03:16:07,407 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 573 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:28,401 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:28,555 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 728 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 557 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:28,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,407 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 558 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:28,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:07,407 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,406 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection502: begin an election at term 558 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,555 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,408 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,434 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection502: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t558, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t558] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t558, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,584 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 729 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:07,408 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,434 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 558 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:28,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,463 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_1 | 2019-10-15 03:16:07,434 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:28,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:07,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,434 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:28,584 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,478 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 574 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:28,613 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:07,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 573 for changeToCandidate datanode_1 | 2019-10-15 03:16:07,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:07,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:07,478 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:07,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection509: begin an election at term 574 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,479 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:07,494 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection509: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t574, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t574] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t574, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,573 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:07,495 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 574 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:07,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:07,573 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 558 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,495 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:07,574 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,495 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:07,586 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection503: begin an election at term 559 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:07,599 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection503: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t559, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t559] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t559, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,591 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 559 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 559 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:07,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:07,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:07,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:07,591 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,694 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 575 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:07,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:16:07,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:07,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:07,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 574 for changeToCandidate datanode_1 | 2019-10-15 03:16:07,694 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:07,726 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | 2019-10-15 03:16:07,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:07,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,692 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection510: begin an election at term 575 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:07,726 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 559 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,721 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection510: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t575, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t575] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t575, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:28,778 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 730 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:07,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 575 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:07,739 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection504: begin an election at term 560 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,778 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:07,768 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection504: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t560, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t560] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t560, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,778 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:07,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:07,768 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 560 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:28,778 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:07,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:07,741 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 560 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:28,833 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:07,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,741 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:07,776 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 576 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,741 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:07,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:07,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,741 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:07,776 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,769 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:07,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,812 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 577 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:07,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 575 for changeToCandidate datanode_1 | 2019-10-15 03:16:07,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:07,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:07,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:07,774 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection511: begin an election at term 576 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,812 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:07,781 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection511: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t576, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t576] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t576, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,826 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:07,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,827 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 560 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:07,827 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,781 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 576 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:07,868 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection505: begin an election at term 561 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,781 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:07,869 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 578 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,781 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,804 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:07,870 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,877 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection505: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t561, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t561] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t561, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,931 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 731 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:07,805 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 576 for changeToCandidate datanode_1 | 2019-10-15 03:16:07,877 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 561 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:07,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:28,931 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:28,932 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:07,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection512: begin an election at term 577 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:28,932 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,824 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection512: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t577, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:28,990 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:07,868 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection512: begin an election at term 578 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:07,890 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 579 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:07,870 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 561 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:07,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:07,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:07,890 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,870 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:08,034 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_2 | 2019-10-15 03:16:07,877 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection512: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t578, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t578] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t578, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:08,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,877 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 578 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:08,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 561 for changeToCandidate datanode_2 | 2019-10-15 03:16:07,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:08,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:07,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:08,038 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 580 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:07,882 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:07,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:08,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:07,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 578 for changeToCandidate datanode_1 | 2019-10-15 03:16:08,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:07,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:08,038 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:07,888 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection513: begin an election at term 579 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection506: begin an election at term 562 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:07,904 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection513: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t579, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t579] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t579, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,053 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection506: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t562, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t562] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t562, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:07,904 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 579 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:08,053 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 562 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:07,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:08,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:07,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:08,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,010 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_1 | 2019-10-15 03:16:08,134 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:08,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:08,134 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 562 for changeToCandidate datanode_2 | 2019-10-15 03:16:08,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 579 for changeToCandidate datanode_3 | 2019-10-15 03:16:29,045 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 732 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:08,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:08,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:29,045 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,159 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection507: begin an election at term 563 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,036 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection514: begin an election at term 580 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,045 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:08,176 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection507: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t563, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t563] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t563, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,042 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 562 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:29,045 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,093 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 733 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:08,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 563 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:08,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:29,093 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:29,094 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:08,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,043 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:08,246 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 581 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:29,094 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:08,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,053 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection514: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t580, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t580] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t580, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,121 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:08,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:08,053 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 580 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:08,247 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:08,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:08,278 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:08,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,278 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:08,278 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 563 for changeToCandidate datanode_2 | 2019-10-15 03:16:08,160 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 563 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:08,278 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:08,161 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:08,292 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection508: begin an election at term 564 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,161 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:08,294 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 582 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:08,161 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,230 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:08,294 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,230 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 580 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:08,309 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection508: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t564, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t564] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t564, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:08,309 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 564 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:08,244 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection515: begin an election at term 581 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:08,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:08,279 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection515: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t581, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t581] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t581, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:08,280 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 581 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:08,431 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 583 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:08,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:08,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,316 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 734 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:08,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:29,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,281 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_1 | 2019-10-15 03:16:08,431 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:29,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:08,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,316 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:08,458 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_2 | 2019-10-15 03:16:08,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 581 for changeToCandidate datanode_3 | 2019-10-15 03:16:29,342 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:08,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:08,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 564 for changeToCandidate datanode_2 | 2019-10-15 03:16:08,292 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection516: begin an election at term 582 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:08,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:08,294 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 564 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:08,471 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection509: begin an election at term 565 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,499 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection509: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t565, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t565] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t565, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:08,294 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:08,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 565 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:08,294 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:08,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:08,309 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection516: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t582, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t582] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t582, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:08,309 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 582 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:08,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:08,309 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:08,525 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 584 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:08,309 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,525 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:08,415 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:16:08,525 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:08,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,526 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:08,416 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 582 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:08,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:08,611 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | 2019-10-15 03:16:08,429 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection517: begin an election at term 583 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:08,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection517: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t583, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t583] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t583, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,383 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:08,446 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 583 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:08,611 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 565 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:08,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:08,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:08,626 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection510: begin an election at term 566 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:08,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:08,473 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 565 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:08,643 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection510: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t566, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t566] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t566, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:08,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,643 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 566 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:08,473 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:08,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:08,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,510 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_1 | 2019-10-15 03:16:08,697 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_2 | 2019-10-15 03:16:08,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:08,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,510 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 583 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:08,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 566 for changeToCandidate datanode_2 | 2019-10-15 03:16:08,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:08,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:08,524 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection518: begin an election at term 584 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:08,713 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection511: begin an election at term 567 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:08,729 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection511: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t567, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t567] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t567, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,541 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection518: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t584, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t584] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t584, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:08,729 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 567 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:08,542 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 584 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:08,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:08,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:08,730 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:08,749 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 585 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:08,628 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 566 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:29,406 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:08,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:08,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:08,750 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,628 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:08,764 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_2 | 2019-10-15 03:16:08,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 567 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:08,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:08,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 567 for changeToCandidate datanode_2 | 2019-10-15 03:16:08,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:08,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:08,714 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:08,770 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection512: begin an election at term 568 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:08,734 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:08,799 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection512: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t568, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:08,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,813 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection512: begin an election at term 569 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:08,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 584 for changeToCandidate datanode_1 | 2019-10-15 03:16:08,841 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection512: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t569, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t569] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t569, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:08,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:08,841 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 569 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:08,748 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection519: begin an election at term 585 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:08,767 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection519: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t585, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t585] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t585, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:08,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:08,767 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 585 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:08,891 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 586 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:08,767 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:08,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:08,767 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:08,771 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 568 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:29,484 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 735 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:08,892 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 587 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:29,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:08,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,771 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,815 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 569 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:08,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:29,507 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 736 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:08,815 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,918 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:29,507 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:08,815 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:08,971 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | 2019-10-15 03:16:29,507 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:08,815 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:08,971 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,507 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:08,876 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_3 | 2019-10-15 03:16:29,530 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:08,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:08,971 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 569 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:08,971 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:08,876 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 585 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:08,984 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection513: begin an election at term 570 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:08,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:09,000 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection513: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t570, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:08,890 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection520: begin an election at term 586 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection513: begin an election at term 571 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:08,894 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection520: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t586, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,048 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection513: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t571, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t571] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t571, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:08,916 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection520: begin an election at term 587 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,048 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 571 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:08,945 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection520: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t587, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t587] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t587, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:08,945 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 587 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:09,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:08,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:09,103 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:08,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:08,986 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 570 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:09,103 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 571 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:08,986 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:09,103 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 588 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:08,986 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:09,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:08,986 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:09,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:09,034 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 571 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:09,104 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:09,117 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection514: begin an election at term 572 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,635 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:09,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,146 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection514: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t572, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t572] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t572, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:09,034 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:09,146 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 572 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:09,088 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_1 | 2019-10-15 03:16:09,146 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:09,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,146 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:09,089 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 587 for changeToCandidate datanode_1 | 2019-10-15 03:16:09,150 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 589 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:09,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:09,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:09,102 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection521: begin an election at term 588 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:09,150 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:09,118 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection521: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t588, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t588] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t588, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:09,317 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:09,118 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 588 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:09,118 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:09,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:09,118 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,317 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 572 for changeToCandidate datanode_2 | 2019-10-15 03:16:09,119 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 572 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:09,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:09,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection515: begin an election at term 573 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:09,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 590 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:09,119 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:09,143 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_1 | 2019-10-15 03:16:09,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:09,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:09,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 588 for changeToCandidate datanode_1 | 2019-10-15 03:16:09,338 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:29,668 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 737 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:09,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:09,364 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection515: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t573, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t573] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t573, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,668 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,147 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection522: begin an election at term 589 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,364 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 573 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:29,668 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:09,176 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection522: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t589, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t589] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t589, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:09,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:29,668 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:09,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:29,673 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:09,464 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | 2019-10-15 03:16:09,176 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 589 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:09,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,176 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:09,176 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:09,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 573 for changeToCandidate datanode_2 | 2019-10-15 03:16:09,310 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:09,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:09,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection516: begin an election at term 574 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:09,310 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 589 for changeToCandidate datanode_1 | 2019-10-15 03:16:09,514 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection516: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t574, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t574] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t574, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:09,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:09,336 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection523: begin an election at term 590 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 573 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:09,514 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 574 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:09,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:09,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:09,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:09,338 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:09,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:09,364 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection523: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t590, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t590] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t590, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:09,515 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 591 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:09,364 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 590 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:09,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:09,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:09,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,515 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:09,488 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 574 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:09,580 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:16:09,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:29,801 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 738 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:09,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,581 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 574 for changeToCandidate datanode_2 | 2019-10-15 03:16:09,488 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:09,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_1 | 2019-10-15 03:16:09,606 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection517: begin an election at term 575 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection517: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t575, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 590 for changeToCandidate datanode_1 | 2019-10-15 03:16:09,632 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection517: begin an election at term 576 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:29,801 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:09,651 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection517: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t576, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t576] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t576, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,513 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection524: begin an election at term 591 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,801 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:09,651 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 576 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:09,552 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection524: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t591, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t591] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t591, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:09,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:09,552 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 591 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:09,552 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:09,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:09,552 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 592 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:09,607 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 575 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:09,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:09,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:09,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,670 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:09,608 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:09,633 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 576 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:09,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,751 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 593 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:09,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,751 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:09,634 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:09,751 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:09,663 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_1 | 2019-10-15 03:16:09,751 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:09,663 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:09,817 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | 2019-10-15 03:16:09,663 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 591 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:09,817 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,663 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:09,817 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 576 for changeToCandidate datanode_2 | 2019-10-15 03:16:09,668 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection525: begin an election at term 592 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,901 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 739 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:09,817 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,676 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection525: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t592, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t592] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t592, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,901 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:09,822 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection518: begin an election at term 577 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:29,902 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:09,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 592 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:09,825 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 594 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:29,902 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:09,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:09,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,007 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:09,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:09,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:09,734 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_1 | 2019-10-15 03:16:09,826 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:09,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:09,879 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection518: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t577, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t577] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t577, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 592 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:09,879 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 577 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:09,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:09,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:09,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection526: begin an election at term 593 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:09,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:09,758 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection526: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t593, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t593] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t593, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:09,924 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 595 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:09,758 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 593 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:09,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:09,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:09,758 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:09,924 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:09,807 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_1 | 2019-10-15 03:16:10,002 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | 2019-10-15 03:16:09,807 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:09,808 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 593 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:10,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:09,808 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:10,002 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 577 for changeToCandidate datanode_2 | 2019-10-15 03:16:09,820 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection527: begin an election at term 594 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:10,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 577 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:10,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection519: begin an election at term 578 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,070 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 740 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,022 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection519: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t578, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:30,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,043 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection519: begin an election at term 579 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,828 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:30,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:10,059 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection519: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t579, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t579] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t579, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,879 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection527: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t594, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t594] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t594, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,070 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:10,059 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 579 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:09,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 594 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:30,211 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:10,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:09,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:10,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:09,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,092 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 596 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:09,908 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:10,092 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,908 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:09,908 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 594 for changeToCandidate datanode_1 | 2019-10-15 03:16:10,092 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:09,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:10,092 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:09,922 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection528: begin an election at term 595 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:09,938 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection528: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t595, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t595] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t595, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:10,180 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:10,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:09,938 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 595 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:10,181 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 579 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:09,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:10,181 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:09,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:10,018 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 578 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,194 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection520: begin an election at term 580 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:10,018 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,222 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection520: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t580, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t580] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t580, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:10,018 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,222 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 580 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:10,018 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:10,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:30,241 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 741 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:10,045 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 579 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:30,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,045 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:10,233 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | 2019-10-15 03:16:10,045 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:10,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,241 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,045 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 580 for changeToCandidate datanode_2 | 2019-10-15 03:16:10,085 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_1 | 2019-10-15 03:16:10,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:30,261 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:10,085 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection521: begin an election at term 581 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,085 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 595 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:10,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection521: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t581, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t581] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t581, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:10,085 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:10,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 581 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:10,091 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection529: begin an election at term 596 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:10,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:10,104 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection529: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t596, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t596] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t596, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:10,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:10,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 596 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:10,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:10,310 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 597 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:10,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:10,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,196 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 580 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:10,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:10,310 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:10,352 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 598 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:10,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,196 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:10,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,248 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 581 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:10,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:10,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,352 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,248 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:10,365 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_2 | 2019-10-15 03:16:10,287 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_3 | 2019-10-15 03:16:30,266 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 742 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,365 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 581 for changeToCandidate datanode_2 | 2019-10-15 03:16:10,287 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 596 for changeToCandidate datanode_3 | 2019-10-15 03:16:30,266 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,365 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:10,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:30,266 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:10,381 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: begin an election at term 582 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,308 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection530: begin an election at term 597 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,266 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:10,412 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t582, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,316 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection530: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t597, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t597] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t597, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,406 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:10,440 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: begin an election at term 583 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,316 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 597 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:10,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t583, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:10,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:10,346 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_1 | 2019-10-15 03:16:10,455 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: begin an election at term 584 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:10,346 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection522: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t584, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t584] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t584, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:10,346 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 597 for changeToCandidate datanode_1 | 2019-10-15 03:16:10,482 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 584 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:10,346 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:10,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:10,350 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection531: begin an election at term 598 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:10,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:10,542 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:10,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:10,542 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 584 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:10,380 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection531: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t598, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t598] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t598, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:10,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:10,380 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 598 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:10,562 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection523: begin an election at term 585 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:10,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:10,564 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 599 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:10,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,383 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 582 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:10,564 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,464 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 743 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,578 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection523: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t585, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t585] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t585, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:30,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,578 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 585 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:10,383 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:30,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:10,578 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:10,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 583 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:30,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:10,578 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,512 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:10,635 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_2 | 2019-10-15 03:16:10,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:10,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,442 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:10,635 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 585 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:10,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:10,652 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection524: begin an election at term 586 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:10,456 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 584 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,667 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection524: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t586, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:10,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,680 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection524: begin an election at term 587 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:10,456 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,696 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection524: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t587, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t587] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t587, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:10,456 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:10,536 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:10,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:10,696 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 587 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:10,536 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 598 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:10,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:10,536 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:10,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,562 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection532: begin an election at term 599 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:10,727 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | 2019-10-15 03:16:10,564 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 585 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:10,727 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:10,728 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 587 for changeToCandidate datanode_2 | 2019-10-15 03:16:10,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:10,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:10,564 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:30,557 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:10,578 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection532: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t599, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t599] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t599, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:10,578 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 599 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:10,742 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection525: begin an election at term 588 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:10,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:10,758 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection525: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t588, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t588] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t588, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,653 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 586 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:10,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 588 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:10,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:10,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:10,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:10,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,654 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:10,761 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 600 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:10,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 587 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:10,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:10,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:10,761 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:10,844 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 601 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:10,744 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 588 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:10,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,744 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:10,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,744 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:10,844 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:10,744 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:10,943 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_2 | 2019-10-15 03:16:10,755 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:10,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:10,943 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 588 for changeToCandidate datanode_2 | 2019-10-15 03:16:10,755 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 599 for changeToCandidate datanode_3 | 2019-10-15 03:16:30,633 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:10,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:10,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:11,293 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: begin an election at term 589 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,760 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection533: begin an election at term 600 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:11,295 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 602 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:10,776 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection533: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t600, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t600] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t600, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,776 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 600 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,776 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:11,295 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:10,776 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,310 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t589, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:10,830 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_1 | 2019-10-15 03:16:11,354 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: begin an election at term 590 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:10,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,367 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t590, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:10,830 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 600 for changeToCandidate datanode_1 | 2019-10-15 03:16:11,374 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: begin an election at term 591 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:10,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:11,376 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 603 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:10,842 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection534: begin an election at term 601 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:10,850 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection534: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t601, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t601] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t601, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:11,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,850 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 601 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:10,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:11,376 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:10,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,383 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection526: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t591, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t591] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t591, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:10,906 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_1 | 2019-10-15 03:16:11,383 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 591 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:11,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:10,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:11,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:10,906 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 601 for changeToCandidate datanode_3 | 2019-10-15 03:16:30,684 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 744 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:10,906 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:11,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 604 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:30,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,293 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection535: begin an election at term 602 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:11,295 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 589 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:30,684 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:30,760 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:11,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:11,295 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:11,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:11,352 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection535: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t602, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t602] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t602, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:11,430 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:11,352 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 602 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:11,463 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:16:11,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:11,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,353 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:11,463 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 591 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:11,355 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 590 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:11,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:11,355 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,477 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection527: begin an election at term 592 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:11,355 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,478 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection527: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t592, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:11,355 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:11,512 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection527: begin an election at term 593 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:11,359 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_1 | 2019-10-15 03:16:11,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 605 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:11,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:11,359 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 602 for changeToCandidate datanode_1 | 2019-10-15 03:16:11,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:11,516 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:11,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:11,517 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection527: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t593, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t593] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t593, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:11,374 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection536: begin an election at term 603 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,778 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:11,517 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 593 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:11,375 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 591 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:11,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:11,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,518 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:11,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,631 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:11,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,375 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:11,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 593 for changeToCandidate datanode_2 | 2019-10-15 03:16:11,383 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection536: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t603, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t603] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t603, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:11,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:11,383 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 603 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:11,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection528: begin an election at term 594 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:11,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:11,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection528: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t594, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t594] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t594, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:11,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,674 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 594 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:11,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:11,409 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:11,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:11,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:11,409 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 603 for changeToCandidate datanode_1 | 2019-10-15 03:16:11,715 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 606 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:11,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:11,428 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection537: begin an election at term 604 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:11,438 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection537: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t604, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t604] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t604, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:11,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,438 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 604 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:11,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:11,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:30,826 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 745 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:11,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,715 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:11,479 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 592 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:30,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,803 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:16:11,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,826 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:11,803 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:11,803 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 594 for changeToCandidate datanode_3 | 2019-10-15 03:16:30,826 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:11,479 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:11,803 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:30,911 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:11,502 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_1 | 2019-10-15 03:16:11,821 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection529: begin an election at term 595 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:11,502 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,844 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection529: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t595, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t595] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t595, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:11,502 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 604 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:11,844 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 595 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:11,502 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:11,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:11,513 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 593 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:11,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:11,513 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:11,848 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 607 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:11,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:11,514 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:11,515 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection538: begin an election at term 605 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:11,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,532 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection538: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t605, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t605] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t605, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:11,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:11,532 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 605 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:11,848 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:11,532 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:11,532 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:11,894 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:11,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:11,894 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 595 for changeToCandidate datanode_2 | 2019-10-15 03:16:11,656 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 594 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:30,958 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:11,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:11,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection530: begin an election at term 596 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:11,911 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection530: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t596, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:11,656 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:11,934 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection530: begin an election at term 597 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:11,656 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:11,953 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection530: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t597, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t597] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t597, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:11,656 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:11,954 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 597 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:11,701 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:11,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:11,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:11,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:11,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 605 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:11,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:12,072 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:11,714 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection539: begin an election at term 606 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:11,725 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection539: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t606, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t606] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t606, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,072 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 597 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:11,725 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 606 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:12,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:11,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:12,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection531: begin an election at term 598 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:11,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,102 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 608 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:11,823 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 595 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,102 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:11,823 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,102 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:11,823 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,102 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:30,983 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 746 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:11,823 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,121 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection531: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t598, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t598] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t598, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:30,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,121 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 598 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:11,841 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | 2019-10-15 03:16:30,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:12,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:11,842 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:30,983 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:11,842 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 606 for changeToCandidate datanode_3 | 2019-10-15 03:16:31,126 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 747 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 609 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:31,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:11,842 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:12,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:31,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:11,846 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection540: begin an election at term 607 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:31,127 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,205 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:11,865 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection540: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t607, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t607] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t607, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,130 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:12,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | 2019-10-15 03:16:11,865 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 607 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:11,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:12,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:11,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,290 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 598 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:11,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 596 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:11,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:12,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:11,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,303 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection532: begin an election at term 599 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:11,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,319 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection532: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t599, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t599] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t599, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:11,936 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 597 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,320 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 599 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:11,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:12,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:11,936 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,382 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:11,936 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,064 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:12,382 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 599 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:12,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:12,064 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 607 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:12,404 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection533: begin an election at term 600 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:12,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:12,421 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection533: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t600, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t600] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t600, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,230 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:12,099 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection541: begin an election at term 608 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,421 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 600 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:12,101 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 598 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:12,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:12,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 610 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:12,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:12,101 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:12,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,121 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection541: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t608, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t608] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t608, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:12,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:12,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 608 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:12,438 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 611 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:12,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:12,438 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,438 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:12,181 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_1 | 2019-10-15 03:16:12,438 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:12,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:12,495 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_2 | 2019-10-15 03:16:12,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 608 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:12,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:12,495 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 600 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,203 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection542: begin an election at term 609 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:12,495 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:12,219 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection542: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t609, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t609] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t609, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,508 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection534: begin an election at term 601 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,268 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 748 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:12,219 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 609 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:12,524 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection534: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t601, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t601] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t601, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,524 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 601 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:31,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:12,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:12,524 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:12,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:31,268 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,524 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,304 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 599 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:31,397 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:12,528 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_2 | 2019-10-15 03:16:12,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:12,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:12,528 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 601 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,304 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:12,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:12,405 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 600 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:12,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,543 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection535: begin an election at term 602 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:12,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection535: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t602, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t602] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t602, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:12,405 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 602 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:12,408 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_1 | 2019-10-15 03:16:12,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:12,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:12,409 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 609 for changeToCandidate datanode_1 | 2019-10-15 03:16:12,609 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 612 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:12,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:12,609 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:12,420 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection543: begin an election at term 610 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,609 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:12,430 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection543: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t610, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,609 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:12,437 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection543: begin an election at term 611 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,665 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:12,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:12,464 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection543: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t611, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t611] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t611, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,665 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 602 for changeToCandidate datanode_3 | 2019-10-15 03:16:31,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 749 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:12,464 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 611 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:12,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:31,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:12,671 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection536: begin an election at term 603 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:12,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 613 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:31,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:12,510 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 601 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:31,479 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:12,672 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:12,510 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:12,690 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection536: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t603, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t603] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t603, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,544 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 602 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:12,690 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 603 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:12,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:12,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:12,690 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,544 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:12,692 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | 2019-10-15 03:16:12,594 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:12,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,594 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:12,692 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 603 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:12,594 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 611 for changeToCandidate datanode_1 | 2019-10-15 03:16:12,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:12,594 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:12,705 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection537: begin an election at term 604 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:12,607 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection544: begin an election at term 612 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,721 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection537: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t604, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t604] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t604, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:12,622 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection544: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t612, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t612] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t612, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,721 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 604 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:12,623 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 612 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:12,721 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:12,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:12,721 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:12,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:12,741 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 614 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:12,656 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_3 | 2019-10-15 03:16:31,566 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:12,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,656 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:12,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,656 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 612 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:12,741 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:12,656 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:12,787 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_2 | 2019-10-15 03:16:12,671 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection545: begin an election at term 613 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:12,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,672 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 603 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:12,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 604 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:12,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:12,672 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:12,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection545: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t613, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t613] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t613, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:12,803 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection538: begin an election at term 605 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,688 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 613 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:12,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 615 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:12,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,706 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 604 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:12,804 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:12,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:12,832 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection538: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t605, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t605] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t605, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,706 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:12,832 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 605 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:12,706 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:12,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:12,727 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:12,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,727 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:31,620 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 750 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,936 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_2 | 2019-10-15 03:16:12,727 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 613 for changeToCandidate datanode_3 | 2019-10-15 03:16:31,620 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,936 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,727 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:31,620 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:12,936 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 605 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,739 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection546: begin an election at term 614 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,620 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,936 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:31,635 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 751 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:12,961 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection539: begin an election at term 606 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,755 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection546: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t614, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t614] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t614, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:12,977 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection539: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t606, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t606] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t606, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,755 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 614 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:31,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:12,977 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 606 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:12,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:31,635 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:12,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:12,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:31,725 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:12,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,792 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:13,008 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | 2019-10-15 03:16:12,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:13,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 614 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:13,009 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 606 for changeToCandidate datanode_2 | 2019-10-15 03:16:12,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:13,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:12,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection547: begin an election at term 615 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:13,022 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection540: begin an election at term 607 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,804 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 605 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:13,041 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection540: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t607, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t607] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t607, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:13,041 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 607 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:12,804 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:12,804 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:12,832 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection547: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t615, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t615] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t615, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:12,832 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 615 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,042 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 616 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:12,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:13,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:12,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:13,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:12,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 606 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:12,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,042 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:12,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,056 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 617 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:12,963 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:31,757 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 752 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:13,024 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 607 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:31,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,056 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:31,757 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:13,024 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,064 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 618 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:13,024 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:31,757 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:13,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,025 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:13,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:13,025 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 615 for changeToCandidate datanode_1 | 2019-10-15 03:16:13,064 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:13,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:13,093 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:13,040 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection548: begin an election at term 616 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:13,048 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection548: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t616, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t616] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t616, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 607 for changeToCandidate datanode_2 | 2019-10-15 03:16:13,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 616 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:13,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:13,107 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection541: begin an election at term 608 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:13,123 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection541: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t608, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t608] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t608, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,050 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:13,124 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 608 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:13,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:13,050 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 616 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:13,124 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:13,124 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:13,055 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection549: begin an election at term 617 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:13,058 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection549: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t617, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,161 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 619 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:13,063 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection549: begin an election at term 618 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,161 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:13,076 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection549: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t618, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t618] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t618, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:31,757 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:13,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 618 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,162 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:31,783 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:13,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:13,251 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:13,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:13,251 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 608 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:13,109 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 608 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:13,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection542: begin an election at term 609 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:13,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,293 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection542: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t609, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t609] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t609, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:13,109 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,294 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 609 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:13,147 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_1 | 2019-10-15 03:16:13,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,147 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:13,147 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 618 for changeToCandidate datanode_1 | 2019-10-15 03:16:13,338 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_2 | 2019-10-15 03:16:13,147 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:13,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:13,160 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection550: begin an election at term 619 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 609 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:13,188 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection550: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t619, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t619] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t619, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:13,189 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 619 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,359 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection543: begin an election at term 610 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:13,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:13,360 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 620 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:13,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:13,267 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 609 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:13,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:13,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,360 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:31,837 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 753 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:13,267 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,377 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection543: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t610, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t610] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t610, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,332 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_3 | 2019-10-15 03:16:31,837 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,377 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 610 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:13,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:31,837 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:13,332 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 619 for changeToCandidate datanode_3 | 2019-10-15 03:16:31,837 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:31,877 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:13,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:13,359 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection551: begin an election at term 620 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:13,469 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 621 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:13,360 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 610 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:13,469 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:13,360 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:13,360 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,469 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:13,361 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,469 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:13,377 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection551: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t620, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t620] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t620, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,483 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:13,377 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 620 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:13,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 610 for changeToCandidate datanode_2 | 2019-10-15 03:16:13,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:13,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,453 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:13,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection544: begin an election at term 611 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:13,513 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection544: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t611, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t611] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t611, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:13,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 611 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:13,453 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 620 for changeToCandidate datanode_1 | 2019-10-15 03:16:13,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:13,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:13,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:13,622 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:16:13,467 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection552: begin an election at term 621 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,990 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 754 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,484 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection552: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t621, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t621] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t621, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:31,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,622 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 611 for changeToCandidate datanode_3 | 2019-10-15 03:16:31,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:13,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 621 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:31,990 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,630 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection545: begin an election at term 612 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:32,020 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 755 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,659 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection545: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t612, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t612] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t612, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,498 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 611 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,659 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 612 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:32,020 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:32,020 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:13,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:32,020 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:13,498 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,662 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_3 | 2019-10-15 03:16:32,076 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 756 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:13,632 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 612 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,076 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,662 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 612 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,076 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:13,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:32,076 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:13,632 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,685 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection546: begin an election at term 613 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,077 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:13,660 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:13,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 622 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:13,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,660 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 621 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:13,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:13,687 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:13,685 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection553: begin an election at term 622 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:13,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 613 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,704 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection546: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t613, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t613] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t613, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:13,704 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 613 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:13,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:13,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:13,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:13,704 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection553: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t622, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t622] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t622, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:13,704 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 622 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:13,704 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:13,717 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | 2019-10-15 03:16:13,704 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:13,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:13,717 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 613 for changeToCandidate datanode_2 | 2019-10-15 03:16:13,723 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 614 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:13,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:13,721 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection547: begin an election at term 614 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:32,105 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:13,728 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection547: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t614, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t614] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t614, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:13,723 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,728 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 614 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:13,750 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_1 | 2019-10-15 03:16:13,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:13,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:13,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,757 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 623 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:13,751 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 622 for changeToCandidate datanode_1 | 2019-10-15 03:16:13,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:13,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:13,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:13,757 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:13,756 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection554: begin an election at term 623 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,767 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:13,772 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection554: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t623, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t623] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t623, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:13,772 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 623 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:13,767 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 614 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:13,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:13,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:13,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,773 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection548: begin an election at term 615 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:13,774 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 615 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,782 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection548: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t615, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t615] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t615, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:13,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:13,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 615 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:13,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:13,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:13,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:32,166 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 757 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:13,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 616 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,797 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_3 | 2019-10-15 03:16:32,166 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:13,797 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 615 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,167 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:13,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,802 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,167 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:13,937 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_1 | 2019-10-15 03:16:13,801 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection549: begin an election at term 616 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,816 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection549: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t616, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t616] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t616, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,226 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:13,937 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 623 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:13,816 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 616 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:13,938 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:13,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:13,961 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection555: begin an election at term 624 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:13,817 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:13,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 617 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:13,916 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:13,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:13,916 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 616 for changeToCandidate datanode_2 | 2019-10-15 03:16:13,963 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:13,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:13,984 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection555: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t624, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t624] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t624, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:13,961 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection550: begin an election at term 617 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:13,985 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 624 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:13,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:13,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:13,962 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 624 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:13,990 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:13,962 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:13,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:13,990 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 624 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:13,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:13,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:13,963 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:14,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection556: begin an election at term 625 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:13,985 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection550: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t617, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t617] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t617, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,022 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection556: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t625, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t625] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t625, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:13,985 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 617 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:14,022 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 625 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:32,289 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:13,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,022 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:13,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:14,022 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 625 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:14,023 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 618 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:14,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:14,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,007 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:14,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,011 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:24ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:14,060 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_1 | 2019-10-15 03:16:14,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:14,011 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 617 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,060 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:14,011 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:14,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection551: begin an election at term 618 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,060 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 625 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:14,036 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection551: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t618, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t618] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t618, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,060 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:14,036 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 618 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:14,065 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection557: begin an election at term 626 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:14,073 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection557: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t626, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t626] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t626, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:14,074 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 626 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:14,036 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,074 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,036 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,074 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:14,132 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 619 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:14,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 626 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:14,132 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,315 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:14,132 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:14,132 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,066 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:14,170 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 620 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,117 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:14,170 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:14,171 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,117 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 618 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:14,171 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:14,178 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_1 | 2019-10-15 03:16:14,130 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection552: begin an election at term 619 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:14,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,147 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection552: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t619, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t619] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t619, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:14,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 626 for changeToCandidate datanode_1 | 2019-10-15 03:16:14,147 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 619 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:14,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:14,147 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:14,184 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection558: begin an election at term 627 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:14,147 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,227 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection558: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t627, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t627] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t627, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:14,155 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:16:14,227 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 627 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:14,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:14,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 619 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,227 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:14,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:14,230 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 621 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:14,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,364 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 758 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:14,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,169 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection553: begin an election at term 620 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,364 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,230 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,365 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:14,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 622 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,185 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection553: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t620, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t620] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t620, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,365 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:14,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 620 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:32,392 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:14,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,326 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:14,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,326 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:14,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 627 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:14,354 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 623 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:14,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,186 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:14,186 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:14,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:14,200 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | 2019-10-15 03:16:14,411 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:14,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:14,200 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 620 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,411 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 627 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:14,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:14,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:14,228 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection554: begin an election at term 621 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,424 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection559: begin an election at term 628 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:14,244 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection554: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t621, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t621] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t621, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:14,443 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection559: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t628, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t628] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t628, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,244 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 621 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:14,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,443 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 628 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:14,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:14,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,310 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:14,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:14,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 624 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,310 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 621 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,393 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 759 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:14,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:14,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:32,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:14,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection555: begin an election at term 622 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,550 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,393 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,339 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection555: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t622, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,614 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | 2019-10-15 03:16:32,540 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:14,353 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection555: begin an election at term 623 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:14,381 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection555: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t623, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t623] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t623, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,614 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 628 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:14,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 623 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:14,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:14,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,627 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 625 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:14,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,627 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:14,426 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 628 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:14,627 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:14,627 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:14,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,629 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection560: begin an election at term 629 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:14,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,642 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection560: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t629, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t629] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t629, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:14,426 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:14,643 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 629 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:14,522 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_2 | 2019-10-15 03:16:14,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:14,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,643 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:14,522 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 623 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,711 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 626 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:14,711 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,548 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection556: begin an election at term 624 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:14,712 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:14,564 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection556: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t624, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t624] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t624, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:14,712 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:14,565 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 624 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:14,752 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 627 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:32,540 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 760 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:32,541 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:14,598 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_3 | 2019-10-15 03:16:32,541 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,753 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,661 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 761 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:14,770 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | 2019-10-15 03:16:14,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 624 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,661 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:32,661 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:14,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 629 for changeToCandidate datanode_1 | 2019-10-15 03:16:14,625 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection557: begin an election at term 625 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,661 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:14,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:14,630 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 629 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:14,784 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection561: begin an election at term 630 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:14,801 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection561: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t630, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t630] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t630, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:14,801 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 630 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:14,630 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:14,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,643 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection557: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t625, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t625] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t625, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:14,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,643 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 625 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:14,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:14,825 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:14,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:14,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:14,684 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_2 | 2019-10-15 03:16:14,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 630 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:14,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:14,684 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 625 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,839 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection562: begin an election at term 631 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:14,867 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection562: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t631, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t631] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t631, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,710 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection558: begin an election at term 626 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:14,867 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 631 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:14,738 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection558: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t626, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t626] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t626, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:14,739 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 626 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:14,867 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:14,867 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:14,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 628 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,742 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:14,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,742 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:14,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:32,661 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,742 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 626 for changeToCandidate datanode_2 | 2019-10-15 03:16:14,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,772 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 762 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,742 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:15,004 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | 2019-10-15 03:16:32,773 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,751 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection559: begin an election at term 627 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,004 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,773 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:15,004 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 631 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,773 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:14,767 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection559: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t627, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t627] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t627, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:32,807 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:15,023 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection563: begin an election at term 632 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,768 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 627 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:15,027 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 629 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:15,027 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:14,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:15,027 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,027 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:14,785 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 630 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:15,039 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection563: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t632, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t632] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t632, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:14,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:15,039 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 632 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:14,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:15,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,785 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:15,039 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:14,840 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 631 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:14,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,056 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:14,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:14,840 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,056 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 632 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:14,896 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:16:15,056 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:14,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,066 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 630 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:14,896 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 627 for changeToCandidate datanode_2 | 2019-10-15 03:16:15,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:14,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:15,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:14,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection560: begin an election at term 628 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,066 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 763 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:14,917 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection560: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t628, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t628] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t628, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,067 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection564: begin an election at term 633 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,980 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:14,917 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 628 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:15,098 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection564: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t633, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t633] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t633, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:15,098 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 633 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:14,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:15,098 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:14,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:15,098 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,014 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:15,225 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_1 | 2019-10-15 03:16:15,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:15,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:15,014 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 628 for changeToCandidate datanode_2 | 2019-10-15 03:16:15,225 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 633 for changeToCandidate datanode_1 | 2019-10-15 03:16:15,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:15,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:15,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 632 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:15,238 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection565: begin an election at term 634 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,025 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection561: begin an election at term 629 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,250 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection565: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t634, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t634] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t634, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:15,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,251 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 634 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:15,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:15,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:15,025 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:15,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,052 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection561: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t629, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:15,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 631 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,065 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection561: begin an election at term 630 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:15,068 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 633 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:15,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:15,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:15,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:32,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,291 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,068 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:32,981 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:15,300 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_1 | 2019-10-15 03:16:15,081 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection561: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t630, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t630] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t630, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:32,981 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:32,996 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 764 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,081 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 630 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:15,300 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 634 for changeToCandidate datanode_3 | 2019-10-15 03:16:32,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:32,996 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:15,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:32,996 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,316 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection566: begin an election at term 635 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,150 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:15,240 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 634 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:15,335 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection566: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t635, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t635] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t635, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:15,335 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 635 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:15,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:15,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:15,240 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,250 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | 2019-10-15 03:16:15,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 632 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,250 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:15,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,250 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 630 for changeToCandidate datanode_2 | 2019-10-15 03:16:15,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:15,443 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,289 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection562: begin an election at term 631 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:15,509 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_1 | 2019-10-15 03:16:15,317 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection562: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t631, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t631] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t631, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:15,509 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,317 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 631 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:15,509 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 635 for changeToCandidate datanode_1 | 2019-10-15 03:16:15,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:15,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:15,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,532 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection567: begin an election at term 636 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,320 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 635 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:16:15,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:15,320 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,548 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection567: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t636, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t636] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t636, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:15,427 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:16:15,548 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 636 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:15,548 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:15,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:15,548 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,427 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 631 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:15,639 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 633 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:15,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection563: begin an election at term 632 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:15,639 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,457 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection563: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t632, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t632] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t632, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:15,639 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,457 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 632 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:33,150 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 765 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:15,680 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_1 | 2019-10-15 03:16:15,457 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:33,151 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,457 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:33,151 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:15,534 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 636 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:15,680 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:33,151 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,680 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 636 for changeToCandidate datanode_3 | 2019-10-15 03:16:33,183 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 766 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:15,680 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:33,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,693 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection568: begin an election at term 637 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,534 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:33,184 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:15,723 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection568: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t637, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t637] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t637, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,184 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,623 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_2 | 2019-10-15 03:16:15,723 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 637 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:33,222 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:15,624 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:15,624 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 632 for changeToCandidate datanode_2 | 2019-10-15 03:16:15,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:15,624 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:15,752 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:15,637 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection564: begin an election at term 633 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:15,665 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection564: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t633, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t633] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t633, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,752 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 637 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:15,665 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 633 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:15,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:15,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:15,757 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection569: begin an election at term 638 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:15,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:15,765 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection569: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t638, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t638] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t638, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,694 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 637 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:15,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 638 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:15,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:15,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:15,695 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:15,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 638 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:15,769 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:15,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,769 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:15,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,769 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 638 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:15,758 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,769 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:15,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 639 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:33,378 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 767 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:33,378 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:15,786 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection570: begin an election at term 639 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,378 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:15,787 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:15,818 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection570: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t639, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t639] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t639, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,378 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:15,808 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | 2019-10-15 03:16:15,818 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 639 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:33,436 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:15,808 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:15,818 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:15,808 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 633 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:15,818 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:15,808 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:15,832 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 634 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,830 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection565: begin an election at term 634 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:15,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection565: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t634, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t634] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t634, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:15,846 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 634 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:15,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:15,832 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:15,969 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:15,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:15,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:15,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:15,969 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 639 for changeToCandidate datanode_1 | 2019-10-15 03:16:15,979 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:15,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:15,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:15,983 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection571: begin an election at term 640 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:15,979 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 634 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:15,985 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 635 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,979 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:15,983 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection566: begin an election at term 635 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:15,985 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 640 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:15,985 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:33,617 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 768 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,000 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection571: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t640, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t640] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t640, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:15,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:16,000 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 640 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:33,617 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:15,985 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,000 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:33,617 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:16,000 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection566: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t635, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t635] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t635, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,000 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:33,651 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:16,000 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 635 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:16,017 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 636 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:16,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:16,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:16,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:16,017 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:16,002 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | 2019-10-15 03:16:16,117 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:16,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,117 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,002 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 635 for changeToCandidate datanode_2 | 2019-10-15 03:16:16,117 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 640 for changeToCandidate datanode_1 | 2019-10-15 03:16:16,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:16,117 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:16,131 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection572: begin an election at term 641 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:16,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection567: begin an election at term 636 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,148 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection572: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t641, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t641] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t641, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:16,045 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection567: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t636, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t636] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t636, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,148 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 641 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:16,045 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 636 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:16,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:16,045 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:16,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:16,045 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:16,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_1 | 2019-10-15 03:16:16,132 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 641 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:16,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:16,194 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 641 for changeToCandidate datanode_1 | 2019-10-15 03:16:16,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:16,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:16,132 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:16,209 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection573: begin an election at term 642 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 642 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:16,236 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection573: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t642, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t642] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t642, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:33,691 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:16,236 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 642 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:16,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:16,210 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:16,220 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:16,237 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 637 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:16,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 636 for changeToCandidate datanode_2 | 2019-10-15 03:16:16,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:16,237 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:16,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:16,330 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:16,236 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection568: begin an election at term 637 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:16,242 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection568: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t637, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t637] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t637, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,330 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 642 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:16,242 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 637 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:16,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:16,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,355 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection574: begin an election at term 643 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:16,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:16,372 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection574: Election REJECTED; received 2 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t643, 749b19c7-0772-44d2-8efd-0664e6aa0748<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t643] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t643, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 643 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:16,372 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 643 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:16,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:16,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:16,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:16,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:16,357 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:16,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 638 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:16,424 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:16,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,424 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:16,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:16,424 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 637 for changeToCandidate datanode_3 | 2019-10-15 03:16:33,790 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 769 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:16,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:33,790 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,523 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_1 | 2019-10-15 03:16:16,424 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:33,791 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:16,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,437 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection569: begin an election at term 638 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,523 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 643 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:16,453 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection569: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t638, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t638] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t638, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:16,453 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 638 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:16,538 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection575: begin an election at term 644 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:16,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,591 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection575 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:16,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:16,539 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 644 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:16,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:16,539 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:16,539 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:16,602 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:16,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:16,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 638 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:16,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:16,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection570: begin an election at term 639 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:16,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection570: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t639, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t639] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t639, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:16,648 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 639 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:16,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:33,791 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:16,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:33,791 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:16,651 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:33,858 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:16,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:16,651 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 639 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:16,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:16,686 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection571: begin an election at term 640 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:16,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 645 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:16,595 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection575: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t644] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t644, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:16,595 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:16,688 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:16,688 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:16,715 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection571: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t640, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t640] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t640, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:16,715 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 640 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:16,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:16,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:16,814 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:16,814 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:16,814 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 640 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:16,814 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:16,828 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection572: begin an election at term 641 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,854 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection572: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t641, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t641] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t641, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:16,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 641 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:16,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:33,861 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 770 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:16,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:33,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:16,908 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 646 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:33,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:16,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:33,862 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:16,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:33,917 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:16,909 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:16,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 647 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:16,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:16,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:16,952 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:16,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 648 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:16,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:16,596 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 644 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:16,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:16,981 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,596 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:17,034 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_2 | 2019-10-15 03:16:16,596 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:17,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,616 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 639 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:17,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 641 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:16,616 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:17,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:16,616 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:17,047 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection573: begin an election at term 642 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:16,617 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,064 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection573: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t642, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t642] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t642, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:33,958 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 771 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:16,642 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_1 | 2019-10-15 03:16:17,064 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 642 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:33,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:33,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:17,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,642 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 644 for changeToCandidate datanode_3 | 2019-10-15 03:16:33,959 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:16,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:34,059 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:16,686 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection576: begin an election at term 645 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:17,087 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:16,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 640 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:17,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:16,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:17,087 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 642 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:16,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:16,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:17,101 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection574: begin an election at term 643 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,696 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection576 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:17,117 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection574: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t643, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t643] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t643, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:17,117 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 643 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:17,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:17,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:17,174 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 649 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:17,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:17,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:17,174 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:17,303 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_3 | 2019-10-15 03:16:34,153 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 772 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:17,303 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:34,153 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:34,153 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:17,303 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 643 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:34,153 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,303 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:34,219 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 773 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:17,308 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 650 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:17,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:34,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:17,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:34,219 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:17,309 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:34,219 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:17,319 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection575: begin an election at term 644 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:34,219 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:17,335 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection575: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t644, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t644] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t644, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:16,715 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection576: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t645] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t645, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:17,335 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 644 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:16,715 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:17,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:17,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:17,368 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 651 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:17,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:17,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:17,368 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:17,448 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 652 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:17,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:17,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:17,448 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:17,528 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:17,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:17,528 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 644 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:17,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:34,358 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 774 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:17,554 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection576: begin an election at term 645 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:34,359 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:17,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 653 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:34,359 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:17,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:34,359 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,559 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:34,433 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:17,559 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:17,571 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection576: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t645, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t645] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t645, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:17,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 645 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:17,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,716 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 645 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:17,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:16,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:17,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 654 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:16,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:17,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,829 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 641 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:17,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:16,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:17,615 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:17,728 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_2 | 2019-10-15 03:16:16,829 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:17,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,891 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:17,728 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 645 for changeToCandidate datanode_2 | 2019-10-15 03:16:16,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:17,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:16,892 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 645 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:17,741 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection577: begin an election at term 646 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:17,769 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection577: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t646, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t646] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t646, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:16,907 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection577: begin an election at term 646 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:34,467 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 775 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:17,769 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 646 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:16,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection577 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:34,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:17,769 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:34,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:17,769 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:34,467 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,821 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 655 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:34,537 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 776 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:17,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:34,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:17,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:34,537 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:17,821 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:34,537 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:17,932 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:34,550 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:17,932 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:17,932 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 646 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:17,932 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:17,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 656 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:17,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:17,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:17,933 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:17,957 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection578: begin an election at term 647 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:17,974 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection578: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t647, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t647] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t647, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:17,974 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 647 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:17,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:17,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:16,932 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection577: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t646] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t646, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:18,076 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_2 | 2019-10-15 03:16:16,932 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:34,584 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 777 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:18,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:34,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:18,076 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 647 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:34,584 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:18,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:34,584 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:18,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection579: begin an election at term 648 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:34,676 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:18,106 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection579: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t648, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t648] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t648, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:18,106 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 648 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:18,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:18,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:18,173 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 657 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:18,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:18,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:18,173 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:18,209 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:18,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:18,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 648 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:18,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:18,234 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection580: begin an election at term 649 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:18,251 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection580: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t649, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t649] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t649, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:18,251 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 649 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:18,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:18,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:34,717 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 778 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:18,293 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | 2019-10-15 03:16:16,933 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 646 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:34,717 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:18,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:34,717 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:16:34,717 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:18,294 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 649 for changeToCandidate datanode_3 | 2019-10-15 03:16:34,738 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 779 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:16,937 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_1 | 2019-10-15 03:16:18,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:34,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:34,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:18,308 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection581: begin an election at term 650 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:18,334 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection581: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t650, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t650] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t650, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:34,738 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:16,937 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 646 for changeToCandidate datanode_1 | 2019-10-15 03:16:18,335 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 650 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:34,796 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:16,937 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:18,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:16,951 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection578: begin an election at term 647 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:18,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:16,956 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection578: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t647, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:18,336 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 658 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:16,979 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection578: begin an election at term 648 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:18,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:16,983 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection578 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:18,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:18,336 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:18,427 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:18,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:18,427 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 650 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:18,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:18,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection582: begin an election at term 651 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:18,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 659 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:18,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:18,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:18,442 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:18,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection582: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t651, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t651] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t651, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:34,915 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 780 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:18,462 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 651 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:34,915 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:18,462 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:34,915 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:18,462 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:34,915 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:18,586 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 660 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:35,008 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:18,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:18,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:18,587 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:18,657 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:18,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:18,657 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 651 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:18,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:18,672 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection583: begin an election at term 652 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:18,700 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection583: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t652, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t652] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t652, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:18,700 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 652 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:17,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection578: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t648] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t648, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:18,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,004 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:18,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:18,798 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:18,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:18,798 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 652 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:18,799 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:18,807 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 661 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:35,159 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 781 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:18,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:35,159 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:18,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:35,159 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:18,807 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:35,159 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:18,807 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection584: begin an election at term 653 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:35,160 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:18,834 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection584: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t653, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t653] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t653, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:18,834 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 653 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:18,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:18,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:18,884 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 662 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:18,884 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:18,884 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:18,884 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:18,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 663 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:18,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:18,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:18,953 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:19,032 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:17,004 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 648 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:19,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:17,004 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:19,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 653 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:19,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 642 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:19,046 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection585: begin an election at term 654 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:19,074 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection585: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t654, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t654] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t654, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:35,185 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:19,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 654 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:17,049 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:17,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 643 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:19,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:19,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:19,089 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 664 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:17,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:19,089 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:19,089 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:17,103 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:19,089 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,138 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_1 | 2019-10-15 03:16:19,187 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:17,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:19,187 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,138 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 648 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:19,187 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 654 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:17,172 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection579: begin an election at term 649 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:19,187 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:17,176 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection579 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:19,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection586: begin an election at term 655 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:19,224 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection586: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t655, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t655] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t655, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:19,225 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 655 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:35,248 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:19,225 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:19,225 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:19,237 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:19,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:19,237 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 655 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:19,237 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:19,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection587: begin an election at term 656 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:19,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 665 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:19,263 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:19,288 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection587: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t656, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t656] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t656, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:19,288 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 656 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:19,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:19,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:19,370 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 666 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:17,198 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection579: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t649] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t649, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:19,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:35,363 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:17,198 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:19,370 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:19,370 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:19,408 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:19,408 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:19,408 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 656 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:19,408 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:19,414 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection588: begin an election at term 657 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:19,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection588: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t657, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t657] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t657, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:19,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 657 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:19,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:19,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:19,486 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:19,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:19,487 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 657 for changeToCandidate datanode_1 | 2019-10-15 03:16:19,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:19,542 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection589: begin an election at term 658 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:19,543 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 667 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:16:19,543 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:19,543 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:19,543 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:19,558 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection589: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t658, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t658] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t658, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:19,559 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 658 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:19,559 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:19,559 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:35,397 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 782 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:19,562 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_3 | 2019-10-15 03:16:35,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:19,562 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:35,397 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:19,562 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 658 for changeToCandidate datanode_3 | 2019-10-15 03:16:35,397 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,199 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 649 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:35,473 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 783 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:19,562 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:35,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:19,576 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection590: begin an election at term 659 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,199 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:19,604 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection590: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t659, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t659] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t659, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:35,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:17,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_3 | 2019-10-15 03:16:35,473 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:19,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 659 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:35,556 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:17,282 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 649 for changeToCandidate datanode_1 | 2019-10-15 03:16:19,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:17,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:19,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:19,767 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | 2019-10-15 03:16:17,306 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection580: begin an election at term 650 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:19,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection580 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:19,767 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 659 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:19,767 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:19,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 668 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:19,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:19,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:19,782 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:19,783 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection591: begin an election at term 660 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:19,796 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection591: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t660, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t660] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t660, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:19,796 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 660 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:19,796 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:19,796 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:19,860 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:19,861 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:35,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 784 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:19,861 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 660 for changeToCandidate datanode_3 | 2019-10-15 03:16:35,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:19,861 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:19,886 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection592: begin an election at term 661 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:35,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:19,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection592: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t661, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t661] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t661, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:35,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:19,902 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 661 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:35,747 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 785 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:35,747 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:19,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:35,747 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:19,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:19,953 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | 2019-10-15 03:16:17,321 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 644 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:19,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:19,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 661 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:17,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:19,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,321 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:17,333 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection580: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t650] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t650, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:19,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection593: begin an election at term 662 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:17,333 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:19,982 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 669 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:19,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:19,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:19,982 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:19,997 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection593: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t662, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:20,000 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection593: begin an election at term 663 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:20,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection593: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t663, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t663] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t663, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:20,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 663 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:20,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:20,061 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:20,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:35,748 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:20,061 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 663 for changeToCandidate datanode_3 | 2019-10-15 03:16:35,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:35,894 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:20,075 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection594: begin an election at term 664 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:20,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection594: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t664, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t664] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t664, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:20,099 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 664 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:20,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:20,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:20,115 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_1 | 2019-10-15 03:16:20,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:20,115 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 664 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:20,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:20,128 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection595: begin an election at term 665 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:20,155 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection595: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t665, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t665] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t665, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,333 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 650 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:20,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 665 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:17,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:20,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:20,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:17,351 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_1 | 2019-10-15 03:16:20,158 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 670 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,351 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:20,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:35,916 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:20,158 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,351 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 650 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,190 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 671 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:17,351 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:20,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,366 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection581: begin an election at term 651 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:20,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:17,371 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection581 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:20,190 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:20,280 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:20,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:20,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 665 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:20,281 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:20,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection596: begin an election at term 666 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:20,322 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 672 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:20,322 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:20,322 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,322 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:20,329 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection596: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t666, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:20,347 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection596: begin an election at term 667 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:35,984 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 786 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,375 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection596: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t667, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t667] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t667, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:35,984 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:20,375 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 667 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:35,984 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,375 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:35,984 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:20,375 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:36,077 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:20,379 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,379 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:20,380 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 667 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:20,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:20,393 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection597: begin an election at term 668 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:20,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection597: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t668, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t668] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t668, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:20,420 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 668 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:20,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:20,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:20,501 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 673 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:17,382 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection581: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t651] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t651, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:20,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:17,382 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:20,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,501 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:20,520 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 674 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:20,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:20,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,520 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:20,521 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:36,174 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 787 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,521 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:36,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:20,521 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 668 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:36,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,521 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:36,174 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:20,549 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection598: begin an election at term 669 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:36,280 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:20,550 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 675 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:20,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:20,551 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:20,575 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection598: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t669, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t669] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t669, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:20,575 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 669 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:20,575 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,575 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:20,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 676 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:20,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:17,383 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 651 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:20,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:17,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:20,599 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:20,634 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | 2019-10-15 03:16:17,431 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:20,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,431 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,634 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 669 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,431 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 651 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:20,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,431 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:20,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection599: begin an election at term 670 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection582: begin an election at term 652 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:36,345 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 788 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection599: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t670, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t670] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t670, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,451 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection582 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:20,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 670 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:36,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:20,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:36,346 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:36,346 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:20,720 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:36,480 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:20,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,720 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 670 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:20,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:20,734 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection600: begin an election at term 671 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:20,750 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection600: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t671, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t671] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t671, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:20,750 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 671 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:20,750 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:20,750 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:20,763 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 677 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:20,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:20,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:20,763 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:20,814 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 678 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:20,814 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,814 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:20,814 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:20,854 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:36,563 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 789 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:36,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:20,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 671 for changeToCandidate datanode_3 | 2019-10-15 03:16:36,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:17,462 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection582: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t652] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t652, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:20,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:36,563 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,462 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:36,593 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 790 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,867 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection601: begin an election at term 672 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:36,594 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:20,883 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection601: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t672, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t672] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t672, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:36,594 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,883 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 672 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:36,594 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:20,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:36,681 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 791 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:20,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:36,681 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:20,968 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 679 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:36,681 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:20,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:36,681 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:20,968 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:20,969 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:36,710 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:21,012 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:21,012 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 672 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:21,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:21,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection602: begin an election at term 673 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:21,033 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection602: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t673, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t673] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t673, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:21,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 673 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:21,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:21,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:21,131 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:21,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:21,131 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 673 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:21,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:21,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection603: begin an election at term 674 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:17,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 652 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:21,137 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 680 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:17,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:21,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:17,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:21,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:36,803 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 792 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:17,549 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_1 | 2019-10-15 03:16:21,137 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:36,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,549 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:21,165 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection603: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t674, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t674] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t674, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:36,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:17,549 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 652 for changeToCandidate datanode_1 | 2019-10-15 03:16:21,165 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 674 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:36,804 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:21,165 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:36,855 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:17,549 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:21,165 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:17,556 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 645 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:21,172 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:7ms, electionTimeout:7ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:17,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:21,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:17,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:21,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 674 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,556 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:21,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,557 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection583: begin an election at term 653 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:21,186 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection604: begin an election at term 675 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:17,562 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection583 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:21,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection604: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t675, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t675] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t675, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:21,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 675 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:21,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:21,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:21,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 681 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:21,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:21,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:21,235 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:21,349 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:21,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:21,349 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 675 for changeToCandidate datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:21,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:36,982 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:21,369 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection605: begin an election at term 676 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:21,398 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection605: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t676, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t676] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t676, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 676 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:21,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:21,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:21,455 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 682 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:21,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:21,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:21,455 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:21,466 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_2 | 2019-10-15 03:16:17,573 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection583: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t653] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t653, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:21,466 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:17,574 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:21,466 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 676 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:21,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection606: begin an election at term 677 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:21,499 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection606: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t677, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t677] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t677, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:21,499 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 677 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:21,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:21,499 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:36,995 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:21,611 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:21,611 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 677 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:21,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:21,615 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 683 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:21,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:21,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection607: begin an election at term 678 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:21,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:21,615 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:21,630 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection607: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t678, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t678] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t678, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:21,630 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 678 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:21,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:21,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:21,737 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 684 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:21,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:21,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:17,574 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 653 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:21,737 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:21,753 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:17,574 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:21,753 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:37,025 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 793 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:17,600 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_1 | 2019-10-15 03:16:21,753 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 678 for changeToCandidate datanode_3 | 2019-10-15 03:16:37,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:21,753 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:37,025 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:17,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 653 for changeToCandidate datanode_1 | 2019-10-15 03:16:21,768 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection608: begin an election at term 679 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,025 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:21,784 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection608: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t679, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t679] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t679, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,189 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:17,613 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection584: begin an election at term 654 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,784 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 679 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:17,618 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection584 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:21,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:21,915 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 685 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:21,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:21,915 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:21,915 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:21,920 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:135ms, electionTimeout:135ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:21,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:21,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 679 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:21,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:21,938 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection609: begin an election at term 680 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:21,964 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection609: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t680, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t680] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t680, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:21,964 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 680 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:21,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:37,238 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 794 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:37,238 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:21,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:21,983 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 686 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:37,238 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:21,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:37,238 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:21,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:37,292 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:21,983 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,083 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:22,083 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:22,083 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 680 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,628 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection584: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t654] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t654, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:22,083 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:17,628 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:22,084 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 687 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:22,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:22,084 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:22,108 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection610: begin an election at term 681 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:22,126 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection610: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t681, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t681] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t681, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:22,126 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 681 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:22,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:22,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:22,158 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:22,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:22,158 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 681 for changeToCandidate datanode_3 | 2019-10-15 03:16:37,292 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 795 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:22,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:37,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:22,172 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection611: begin an election at term 682 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,292 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:22,200 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection611: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t682, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t682] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t682, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,292 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:22,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 682 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:37,450 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:22,201 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:22,213 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:22,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:22,213 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 682 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:22,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:22,227 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection612: begin an election at term 683 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:22,242 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection612: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t683, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t683] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t683, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:22,242 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 683 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:17,629 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 654 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:22,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:22,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:17,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:22,298 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 688 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:17,743 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 646 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:22,299 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,743 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:17,743 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:22,299 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:17,743 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:22,299 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:22,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 689 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:17,801 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_1 | 2019-10-15 03:16:22,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:37,463 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:17,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:17,801 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 654 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:22,436 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:17,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:22,442 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_2 | 2019-10-15 03:16:17,816 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection585: begin an election at term 655 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:22,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,825 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection585 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:22,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 683 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:22,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:22,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection613: begin an election at term 684 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,479 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection613: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t684, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t684] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t684, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:22,479 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 684 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:22,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:22,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:22,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 690 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:22,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:22,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:22,514 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:22,677 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:22,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:37,466 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 796 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:22,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 684 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:37,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:22,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:37,466 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:22,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection614: begin an election at term 685 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,466 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:22,696 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 691 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:37,528 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 797 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:22,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:37,529 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:22,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:37,529 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:22,696 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:37,529 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:22,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection614: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t685, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t685] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t685, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:37,541 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:22,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 685 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:22,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,835 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection585: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t655] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t655, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:17,835 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:22,731 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:22,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:22,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 685 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:22,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:22,757 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection615: begin an election at term 686 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:22,774 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection615: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t686, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t686] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t686, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 686 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:22,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:22,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:22,909 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 692 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:22,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:22,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:22,909 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:22,944 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:22,944 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:22,944 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 686 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:37,619 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 798 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:22,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:37,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:22,957 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection616: begin an election at term 687 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:37,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:22,982 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection616: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t687, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t687] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t687, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:37,619 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:22,982 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 687 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:37,684 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:22,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:22,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:23,022 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:23,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:17,836 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 655 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:23,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 687 for changeToCandidate datanode_2 | 2019-10-15 03:16:17,836 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:23,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:17,836 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:17,915 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_1 | 2019-10-15 03:16:23,048 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection617: begin an election at term 688 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:17,915 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:23,066 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection617: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t688, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t688] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t688, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:17,915 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 655 for changeToCandidate datanode_1 | 2019-10-15 03:16:23,067 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 688 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:23,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,916 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:23,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:17,932 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection586: begin an election at term 656 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:23,077 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 693 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:17,938 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection586 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:23,077 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:23,077 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:23,077 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:23,165 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:23,165 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:37,730 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 799 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:23,165 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 688 for changeToCandidate datanode_3 | 2019-10-15 03:16:37,730 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:23,165 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:37,730 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,177 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection618: begin an election at term 689 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,730 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection618: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t689, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t689] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t689, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:37,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 800 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 689 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:37,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:37,815 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 694 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:23,193 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:23,327 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 695 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:23,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:23,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:17,958 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection586: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t656] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t656, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:23,327 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:17,958 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:23,359 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:23,360 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 689 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:23,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:23,362 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 696 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:23,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:37,815 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:23,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:37,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:23,362 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,363 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection619: begin an election at term 690 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:37,900 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 801 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:23,392 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection619: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t690, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t690] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t690, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:37,900 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:23,392 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 690 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:37,900 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:23,392 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:37,901 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:23,392 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:23,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_3 | 2019-10-15 03:16:37,935 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 802 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:23,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:37,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:23,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 690 for changeToCandidate datanode_3 | 2019-10-15 03:16:37,936 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:23,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:37,936 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:23,450 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 697 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:38,018 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:23,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:23,450 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:23,450 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection620: begin an election at term 691 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:17,959 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 656 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:23,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection620: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t691, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t691] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t691, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:17,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:23,476 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 691 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:17,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:23,476 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:17,960 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 647 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:23,476 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:17,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:23,550 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_2 | 2019-10-15 03:16:17,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:23,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:17,960 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:23,550 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 691 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:18,091 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 648 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:23,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:18,092 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:23,575 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection621: begin an election at term 692 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:18,092 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:23,602 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection621: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t692, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t692] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t692, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:18,092 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:23,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 692 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:18,157 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:23,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:18,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:18,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 656 for changeToCandidate datanode_1 | 2019-10-15 03:16:23,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:38,049 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 803 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:18,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:23,658 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_3 | 2019-10-15 03:16:38,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,171 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection587: begin an election at term 657 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:23,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:38,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:23,658 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 692 for changeToCandidate datanode_2 | 2019-10-15 03:16:18,176 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection587 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:38,050 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:23,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,675 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection622: begin an election at term 693 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:23,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 698 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:23,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:23,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:23,677 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,691 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection622: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t693, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t693] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t693, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:23,691 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 693 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:23,691 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:23,691 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:23,828 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 699 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:23,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:23,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:23,828 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:23,881 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | 2019-10-15 03:16:38,050 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:23,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:38,119 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:23,881 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 693 for changeToCandidate datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:23,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:23,894 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection623: begin an election at term 694 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,185 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection587: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t657] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t657, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:23,922 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection623: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t694, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t694] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t694, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:18,185 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:23,922 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 694 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:23,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,922 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:23,938 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:23,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:23,939 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 694 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:23,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:23,962 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection624: begin an election at term 695 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:23,967 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 700 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:23,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:23,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:23,967 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:24,012 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection624: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t695, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t695] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t695, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:24,012 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 695 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:24,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:38,156 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 804 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:24,012 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:38,156 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,099 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:38,156 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:24,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:38,156 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,099 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 695 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:24,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:38,279 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 805 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:24,125 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: begin an election at term 696 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:38,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,138 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t696, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:38,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:24,142 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: begin an election at term 697 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:38,279 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,152 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t697, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:38,310 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:18,186 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 657 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:24,159 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: begin an election at term 698 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:18,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:24,174 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection625: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t698, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t698] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t698, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:24,175 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 698 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,236 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 649 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:24,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:18,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:18,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:24,196 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 701 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:24,196 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,236 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:18,308 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_1 | 2019-10-15 03:16:24,196 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:18,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,197 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:24,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 702 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:18,308 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 657 for changeToCandidate datanode_1 | 2019-10-15 03:16:24,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:18,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:24,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:18,310 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 650 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:24,235 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:18,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 703 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:18,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:24,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:18,310 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:18,334 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection588: begin an election at term 658 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:24,262 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:38,412 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:18,338 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection588 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:24,366 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:24,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:24,366 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 698 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:24,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:24,380 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection626: begin an election at term 699 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:24,397 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection626: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t699, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t699] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t699, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:24,397 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 699 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:24,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:24,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:24,441 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 704 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:24,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:24,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:24,441 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:24,566 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:38,507 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 806 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:24,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:38,507 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,566 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 699 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:38,507 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:24,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:38,507 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,581 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection627: begin an election at term 700 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:38,582 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:24,597 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection627: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t700, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t700] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t700, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:24,597 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 700 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:24,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,350 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection588: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t658] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t658, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:24,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:18,350 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:24,620 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 705 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:24,620 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:24,620 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:24,620 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:24,696 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:24,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:24,696 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 700 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:24,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:24,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection628: begin an election at term 701 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:24,719 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection628: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t701, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t701] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t701, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:24,720 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 701 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:24,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:38,667 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 807 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:24,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:38,667 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:24,805 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 706 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:38,667 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:24,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:38,667 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:38,698 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:24,805 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:24,867 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:24,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:24,867 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 701 for changeToCandidate datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:24,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:18,350 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 658 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:24,882 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection629: begin an election at term 702 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:24,883 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 707 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:18,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:24,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,425 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:18,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:18,425 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 658 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:24,883 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:18,441 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection589: begin an election at term 659 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:24,883 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:18,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 651 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:24,899 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection629: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t702, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t702] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t702, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:24,899 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 702 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:18,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:24,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:38,724 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:24,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:18,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:24,982 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:18,442 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:24,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection589 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:24,982 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 702 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:24,982 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:25,009 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection630: begin an election at term 703 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:25,010 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 708 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:25,010 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:25,010 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:25,010 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:25,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection630: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t703, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t703] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t703, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:25,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 703 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:25,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:25,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:25,217 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:25,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:25,217 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 703 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:25,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:38,826 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:25,232 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection631: begin an election at term 704 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:25,234 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 709 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:25,234 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:25,248 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection631: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t704, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t704] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t704, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:25,248 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 704 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:25,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:18,462 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection589: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t659] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t659, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:25,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,462 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:25,375 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:25,375 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:25,375 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 704 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:25,375 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:25,398 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection632: begin an election at term 705 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:25,400 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 710 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:38,843 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 808 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:38,843 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:38,844 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:38,844 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:25,400 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:25,426 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection632: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t705, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t705] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t705, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:39,005 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:25,426 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 705 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:25,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:25,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:25,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 711 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_1 | 2019-10-15 03:16:25,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:25,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:25,460 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:25,501 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:25,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:25,501 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 705 for changeToCandidate datanode_2 | 2019-10-15 03:16:18,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 659 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:25,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:18,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:25,506 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection633: begin an election at term 706 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:25,514 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection633: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t706, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t706] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t706, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,560 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:25,514 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 706 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:18,560 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:18,560 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 659 for changeToCandidate datanode_1 | 2019-10-15 03:16:25,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:18,560 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:25,514 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:18,585 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection590: begin an election at term 660 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:25,537 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:18,589 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection590 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:39,005 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 809 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:25,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:39,005 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:25,537 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 706 for changeToCandidate datanode_3 | 2019-10-15 03:16:39,005 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:25,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:39,005 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:25,541 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection634: begin an election at term 707 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:39,105 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:25,549 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection634: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t707, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t707] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t707, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:25,549 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 707 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:25,549 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:25,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:25,675 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 712 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:25,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:25,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:25,675 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:25,738 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:25,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:25,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 707 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:25,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:25,752 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection635: begin an election at term 708 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:25,779 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection635: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t708, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t708] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t708, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:39,174 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 810 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:25,779 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 708 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:25,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:25,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:39,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection590: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t660] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t660, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:25,780 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 713 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:39,174 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:18,611 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:39,174 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:25,780 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:39,200 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 811 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:25,780 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:39,200 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:25,780 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:39,200 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:25,872 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 714 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:39,200 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:25,872 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:39,215 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:25,872 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:25,872 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:25,902 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:25,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:25,902 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 708 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:25,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:25,916 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection636: begin an election at term 709 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:25,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 715 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:25,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:25,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:25,918 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:25,933 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection636: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t709, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t709] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t709, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:25,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 709 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:25,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:25,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:25,978 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 716 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:25,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:40,026 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 812 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:25,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 660 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:25,978 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:18,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:25,999 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_3 | 2019-10-15 03:16:40,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:25,999 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,674 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 652 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:26,000 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 709 for changeToCandidate datanode_3 | 2019-10-15 03:16:40,026 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:18,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:26,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:18,674 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:18,674 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:26,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection637: begin an election at term 710 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,017 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection637: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t710, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:26,029 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection637: begin an election at term 711 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 660 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:26,044 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection637: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t711, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t711] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t711, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:26,044 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 711 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:18,805 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection591: begin an election at term 661 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:26,044 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:18,809 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 653 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:26,044 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:18,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:26,177 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:18,809 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:26,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:18,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection591 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:26,177 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 711 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:26,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:26,211 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection638: begin an election at term 712 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:26,212 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 717 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,213 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:26,239 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection638: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t712, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t712] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t712, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:26,239 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 712 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:40,026 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,239 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:40,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:26,239 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:40,055 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:26,329 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 718 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:26,329 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,329 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:26,329 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:26,342 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:26,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:26,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 712 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:26,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:26,355 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection639: begin an election at term 713 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:26,383 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection639: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t713, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t713] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t713, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:26,383 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 713 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:26,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:26,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,413 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 719 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:26,413 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:18,834 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection591: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t661] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t661, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:26,413 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,834 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:26,413 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,421 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_3 | 2019-10-15 03:16:40,074 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 813 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:26,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:26,421 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 713 for changeToCandidate datanode_3 | 2019-10-15 03:16:40,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:26,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:40,074 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:26,440 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection640: begin an election at term 714 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:40,074 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:40,087 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 814 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:26,455 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection640: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t714, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:40,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:26,467 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection640: begin an election at term 715 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:40,087 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:26,483 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection640: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t715, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t715] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t715, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:40,087 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:26,483 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 715 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:40,087 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:26,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,483 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:26,572 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:26,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:26,572 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 715 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:26,572 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:26,586 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection641: begin an election at term 716 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:26,614 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection641: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t716, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t716] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t716, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:26,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 716 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:26,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:18,834 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 661 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:26,636 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 720 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:18,835 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:26,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,835 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:26,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,637 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:18,868 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_1 | 2019-10-15 03:16:26,686 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_3 | 2019-10-15 03:16:40,152 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:18,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:26,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:18,868 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 661 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:18,868 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:18,883 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection592: begin an election at term 662 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:26,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 716 for changeToCandidate datanode_2 | 2019-10-15 03:16:18,888 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection592 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:26,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:26,714 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection642: begin an election at term 717 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:26,716 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 721 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:26,716 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:26,740 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection642: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t717, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t717] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t717, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:26,740 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 717 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:26,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:26,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 722 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:26,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:40,322 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 815 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:26,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:40,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:26,865 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:40,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:26,884 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_3 | 2019-10-15 03:16:40,322 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:40,344 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:26,884 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:26,884 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 717 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:26,884 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:26,898 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection643: begin an election at term 718 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,906 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection592: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t662] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t662, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:26,913 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection643: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t718, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t718] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t718, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:18,906 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:26,913 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 718 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:26,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:26,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:27,002 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 723 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:27,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:27,002 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:27,002 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:27,022 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:27,022 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:27,022 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 718 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:27,022 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,028 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 724 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:40,427 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:27,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:27,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:27,028 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:27,030 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection644: begin an election at term 719 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:27,047 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection644: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t719, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t719] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t719, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:27,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 719 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:27,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:27,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:27,048 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 725 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:27,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:18,907 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 662 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:27,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,907 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:27,048 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:18,907 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:27,176 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | 2019-10-15 03:16:18,946 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:27,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:18,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:27,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 719 for changeToCandidate datanode_2 | 2019-10-15 03:16:18,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 662 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:27,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:18,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:18,952 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection593: begin an election at term 663 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,190 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection645: begin an election at term 720 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,958 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection593 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:40,496 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 816 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:27,196 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 726 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:40,496 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:40,497 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:27,196 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:40,497 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:27,196 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:40,533 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:27,196 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:27,199 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection645: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t720, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t720] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t720, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:27,199 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 720 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:27,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:27,199 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:27,318 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 727 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:27,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:27,318 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:27,318 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:27,382 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:27,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:27,382 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 720 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:27,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:27,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection646: begin an election at term 721 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,970 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection593: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t663] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t663, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,414 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection646: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t721, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t721] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t721, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:18,970 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:27,414 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 721 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:40,659 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 817 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:27,414 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:40,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:27,414 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:40,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:27,472 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 728 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:40,659 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:27,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:40,686 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:27,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:27,472 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,571 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:27,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:27,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 721 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:27,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:27,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection647: begin an election at term 722 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:27,641 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection647: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t722, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t722] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t722, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:27,641 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 722 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:27,641 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:27,641 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:27,644 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 729 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:27,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:27,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:27,644 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:27,768 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 730 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:40,826 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:27,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:18,970 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 663 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:27,769 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:18,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:27,774 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:18,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:27,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:19,048 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 654 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:27,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 722 for changeToCandidate datanode_2 | 2019-10-15 03:16:19,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:27,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:19,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:27,794 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection648: begin an election at term 723 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,048 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:19,073 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_1 | 2019-10-15 03:16:27,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection648: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t723, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t723] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t723, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:19,073 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:27,800 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 723 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:19,073 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 663 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:27,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:19,073 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:27,801 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,087 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection594: begin an election at term 664 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:27,862 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 731 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:19,092 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection594 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:27,862 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:27,862 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:27,862 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:27,890 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 732 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:27,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:40,861 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 818 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:27,890 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:40,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:27,991 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_3 | 2019-10-15 03:16:40,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:27,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:40,861 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:27,991 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 723 for changeToCandidate datanode_3 | 2019-10-15 03:16:40,997 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:27,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,004 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection649: begin an election at term 724 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:28,007 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection649: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t724, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:28,031 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection649: begin an election at term 725 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:28,059 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection649: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t725, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t725] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t725, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:28,060 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 725 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:28,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:28,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:28,062 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 733 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:28,062 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:28,062 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:28,062 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:28,190 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 734 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:19,102 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection594: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t664] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t664, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:28,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:19,102 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:28,190 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,190 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:41,023 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 819 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:28,191 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:28,191 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:41,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:28,191 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 725 for changeToCandidate datanode_3 | 2019-10-15 03:16:41,023 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:41,023 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:28,191 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:28,203 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection650: begin an election at term 726 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,061 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,218 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection650: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t726, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t726] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t726, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:28,219 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 726 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:28,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:28,219 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:28,323 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 735 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:28,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:28,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:28,323 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:28,389 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:28,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:28,389 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 726 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:28,389 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:28,398 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection651: begin an election at term 727 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:28,400 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 736 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:28,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:28,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:28,400 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:41,070 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:28,425 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection651: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t727, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t727] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t727, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 664 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,425 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 727 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:19,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:28,425 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:19,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:28,425 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,203 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 655 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:28,540 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | 2019-10-15 03:16:19,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:28,540 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,203 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:28,540 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 727 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:28,540 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:19,203 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:28,554 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection652: begin an election at term 728 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:19,236 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_1 | 2019-10-15 03:16:28,574 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection652: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t728, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:19,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:28,583 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection652: begin an election at term 729 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:19,236 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 664 for changeToCandidate datanode_1 | 2019-10-15 03:16:28,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection652: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t729, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t729] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t729, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:19,237 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:28,609 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 729 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:19,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection595: begin an election at term 665 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:28,609 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:19,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 656 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:28,609 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:28,612 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 737 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:19,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:41,104 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 820 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:28,612 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:41,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:28,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,265 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection595 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:41,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:28,613 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:41,104 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:28,762 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:41,121 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 821 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:28,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:41,121 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:16:41,121 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:41,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:28,763 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 729 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:41,207 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:28,776 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection653: begin an election at term 730 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:28,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection653: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t730, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t730] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t730, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:28,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 730 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:28,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:28,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:28,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 738 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:28,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:28,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:28,836 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:28,915 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:28,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:28,916 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 730 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:28,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:28,929 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection654: begin an election at term 731 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:28,951 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection654: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t731, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t731] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t731, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,229 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 822 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:28,951 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 731 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:19,288 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection595: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t665] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t665, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:28,951 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:19,288 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:41,229 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:28,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:41,229 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:28,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 739 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:41,304 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:28,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:28,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:28,990 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:29,027 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:29,028 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 731 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:29,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:29,043 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection655: begin an election at term 732 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:29,059 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection655: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t732, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t732] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t732, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:29,059 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 732 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:29,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:29,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:29,076 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:29,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,076 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 732 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:29,076 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:29,091 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection656: begin an election at term 733 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:41,364 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:29,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection656: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t733, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t733] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t733, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,120 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 733 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:29,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:29,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 665 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:29,121 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 740 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:19,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:29,121 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:19,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:29,121 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:19,352 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_1 | 2019-10-15 03:16:29,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:19,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:19,352 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 665 for changeToCandidate datanode_1 | 2019-10-15 03:16:29,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 733 for changeToCandidate datanode_2 | 2019-10-15 03:16:19,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:29,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:19,368 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection596: begin an election at term 666 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:29,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection657: begin an election at term 734 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:19,373 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection596 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:29,342 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection657: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t734, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t734] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t734, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 734 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 741 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:41,458 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 823 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:29,342 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:41,458 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:41,458 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:29,383 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 742 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:29,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:29,383 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:29,383 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,406 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 743 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:29,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:29,407 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:29,407 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:29,458 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:29,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:29,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 734 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:29,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:29,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection658: begin an election at term 735 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:29,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection658: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t735, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:19,375 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection596: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t666] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t666, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:29,506 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection658: begin an election at term 736 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,375 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:41,459 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,529 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection658: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t736, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t736] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t736, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:29,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 736 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:41,605 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:29,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:29,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:29,530 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 744 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:29,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:29,530 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:29,634 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 745 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:29,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:29,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:29,635 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:29,651 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:29,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:29,651 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 736 for changeToCandidate datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:29,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:29,667 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection659: begin an election at term 737 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:29,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 746 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:29,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:41,668 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 824 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:29,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:41,668 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,671 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:41,668 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:19,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 666 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:29,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection659: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t737, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t737] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t737, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,668 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:19,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:29,674 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 737 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:41,820 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:19,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:29,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:19,415 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 657 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:29,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:29,784 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:16:19,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,416 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:29,784 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 737 for changeToCandidate datanode_2 | 2019-10-15 03:16:19,520 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:19,520 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:29,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection660: begin an election at term 738 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,520 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 666 for changeToCandidate datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:29,801 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 747 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,520 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:19,542 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection597: begin an election at term 667 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:19,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 658 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:29,801 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:19,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:29,826 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection660: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t738, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t738] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t738, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:19,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:29,826 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 738 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:19,543 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:29,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:19,546 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection597 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:29,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,878 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_3 | 2019-10-15 03:16:41,880 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:29,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,878 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 738 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:29,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:29,900 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection661: begin an election at term 739 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:29,917 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection661: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t739, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t739] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t739, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:29,917 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 739 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:29,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:29,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:30,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 748 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:30,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:30,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:30,007 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:30,054 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:30,054 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:30,054 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 739 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:30,054 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:30,068 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection662: begin an election at term 740 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:30,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection662: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t740, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t740] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t740, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:41,894 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 825 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:30,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 740 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:19,558 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection597: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t667] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t667, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:30,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:41,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,559 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:30,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:41,894 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,211 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 749 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:41,894 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:30,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:42,009 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:30,211 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:30,211 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:30,224 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:30,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:30,224 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 740 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:30,224 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:30,239 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection663: begin an election at term 741 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:30,248 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection663: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t741, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t741] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t741, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:30,248 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 741 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:30,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:30,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:30,260 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:30,260 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:30,260 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 741 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:30,260 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:30,261 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 750 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:42,012 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 826 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:30,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:42,012 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,559 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 667 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:30,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:42,012 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:19,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:42,012 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:30,261 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:19,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:42,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 827 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:19,577 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 659 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:42,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:42,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:30,264 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection664: begin an election at term 742 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:42,041 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:30,273 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection664: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t742, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t742] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t742, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,577 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:42,144 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:30,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 742 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:19,754 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_1 | 2019-10-15 03:16:30,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:19,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,754 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 667 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:30,406 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 751 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:30,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,755 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:30,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,780 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection598: begin an election at term 668 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:30,406 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:19,785 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 660 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:30,448 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:16:19,785 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:30,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,785 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:30,449 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 742 for changeToCandidate datanode_2 | 2019-10-15 03:16:19,785 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:30,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:19,785 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection598 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:30,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection665: begin an election at term 743 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:30,493 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection665: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t743, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t743] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t743, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:30,494 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 743 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:30,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:30,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:30,512 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 752 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:42,221 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:30,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:30,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,556 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 753 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:30,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:30,556 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:30,557 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:30,633 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 754 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:30,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:30,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:30,633 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:30,669 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:30,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:30,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 743 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:30,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:30,682 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection666: begin an election at term 744 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,795 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection598: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t668] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t668, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:30,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection666: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t744, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t744] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t744, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:19,796 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:42,290 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:30,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 744 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:30,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:30,760 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 755 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:30,760 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:30,760 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:30,760 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:30,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 756 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:30,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:30,778 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:30,818 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:30,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:30,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 744 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:30,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:30,824 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection667: begin an election at term 745 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:30,840 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection667: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t745, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t745] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t745, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:42,315 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 828 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:30,840 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 745 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:42,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:30,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:42,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:30,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:42,315 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:30,911 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 757 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | 2019-10-15 03:16:42,470 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:19,796 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 668 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:30,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:30,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,796 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:30,911 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:30,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 758 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:30,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,796 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:30,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:19,888 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 661 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:30,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:19,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:19,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:30,977 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:19,888 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:30,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,972 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:19,972 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:19,972 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 668 for changeToCandidate datanode_1 | 2019-10-15 03:16:30,977 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 745 for changeToCandidate datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:19,972 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:30,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:30,982 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection668: begin an election at term 746 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:19,979 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 662 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:30,989 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection668: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t746, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t746] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t746, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:42,480 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:19,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:30,989 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 746 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:19,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:30,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:19,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:30,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:19,981 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection599: begin an election at term 669 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:31,105 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:19,990 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection599 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:31,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:31,105 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 746 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:31,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:31,125 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection669: begin an election at term 747 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:31,129 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: changes role from FOLLOWER to FOLLOWER at term 759 for recognizeCandidate:749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:31,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,130 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:31,130 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:31,143 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection669: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t747, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t747] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t747, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:31,144 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 747 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:31,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:42,508 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 829 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:31,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:42,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:31,173 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove FOLLOWER 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7:t759, leader=null, voted=null, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f0,i0, conf=0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_3 | 2019-10-15 03:16:42,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:31,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: shutdown datanode_3 | 2019-10-15 03:16:42,508 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:31,177 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:42,699 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 830 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:31,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:42,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:31,177 INFO impl.StateMachineUpdater: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-StateMachineUpdater: set stopIndex = -1 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:31,177 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:31,178 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7: closes. applyIndex: -1 datanode_3 | 2019-10-15 03:16:42,699 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:31,180 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:31,183 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-B866F3D9F4B7-SegmentedRaftLogWorker close() datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:31,204 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove LEADER 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B:t1, leader=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLog:OPENED:c0,f0,i0, conf=0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null RUNNING datanode_2 | 2019-10-15 03:16:19,998 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection599: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t669] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t669, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:31,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: shutdown datanode_2 | 2019-10-15 03:16:19,998 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:31,204 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-6E9FEAD04B3B,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:31,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:31,205 INFO impl.PendingRequests: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-PendingRequests: sendNotLeaderResponses datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:31,205 INFO impl.StateMachineUpdater: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-StateMachineUpdater: set stopIndex = 0 datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:31,206 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B: closes. applyIndex: 0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:31,206 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:31,207 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-6E9FEAD04B3B-SegmentedRaftLogWorker close() datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:31,231 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:42,700 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:42,700 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:42,725 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 831 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:42,725 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:42,725 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:31,242 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:97ms datanode_2 | 2019-10-15 03:16:19,998 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 669 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:31,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:19,998 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:31,242 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 747 for changeToCandidate datanode_2 | 2019-10-15 03:16:19,998 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:31,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:20,002 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 663 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:31,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection670: begin an election at term 748 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:31,290 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection670: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t748, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:20,002 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:31,395 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection670: begin an election at term 749 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:20,077 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 664 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:31,397 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:20,077 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,077 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:20,077 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:42,725 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:20,129 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 665 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:42,725 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:42,809 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,129 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,150 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,151 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 669 for changeToCandidate datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,151 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:20,156 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection600: begin an election at term 670 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:20,161 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection600 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:31,421 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection670: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t749, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t749] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t749, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:31,421 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 749 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:31,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:31,421 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:31,479 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:42,879 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 832 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:42,879 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:42,879 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:42,879 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:42,944 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,173 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection600: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t670] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t670, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,173 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:31,565 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:43,057 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:31,610 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:31,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:31,610 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 749 for changeToCandidate datanode_2 | 2019-10-15 03:16:20,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 670 for DISCOVERED_A_NEW_TERM datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:20,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:43,123 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 833 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:31,617 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection671: begin an election at term 750 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:43,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:31,627 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection671: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t750, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:43,123 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:20,185 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_1 | 2019-10-15 03:16:31,633 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection671: begin an election at term 751 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:43,123 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:31,643 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection671: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t751, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t751] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t751, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,185 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:43,241 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:20,185 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 670 for changeToCandidate datanode_1 | 2019-10-15 03:16:31,643 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 751 for DISCOVERED_A_NEW_TERM datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:20,185 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:31,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:20,188 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection601: begin an election at term 671 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:31,643 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:31,725 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,194 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection601 got exception when requesting votes: {} datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:43,311 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 834 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:43,311 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:31,737 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_3 | 2019-10-15 03:16:43,311 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:31,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:31,737 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 751 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:31,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:31,755 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection672: begin an election at term 752 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection601: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t671] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t671, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:31,757 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:20,195 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:43,311 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:43,311 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:43,445 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,780 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection672: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t752, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t752] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t752, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:31,781 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 752 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:31,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:31,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:20,195 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 671 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:31,783 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:20,195 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:20,195 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:20,304 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:20,304 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 671 for changeToCandidate datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,320 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection602: begin an election at term 672 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:43,508 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,322 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 666 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,322 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,325 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection602 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:31,822 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:31,822 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:31,822 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 752 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:31,822 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:31,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection673: begin an election at term 753 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,864 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection673: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t753, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t753] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t753, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 753 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:43,538 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 835 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:31,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:43,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:31,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:31,877 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:43,538 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:43,538 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:43,572 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,348 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection602: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t672] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t672, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,348 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:31,973 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:31,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:43,645 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 836 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:31,973 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 753 for changeToCandidate datanode_3 | 2019-10-15 03:16:43,645 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:43,645 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:31,973 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:43,645 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:43,672 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 837 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:31,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection674: begin an election at term 754 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:43,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:32,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection674: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t754, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:43,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:32,018 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection674: begin an election at term 755 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:43,672 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:32,037 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection674: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t755, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t755] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t755, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:43,722 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 838 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:32,037 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 755 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:43,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:32,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:32,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:43,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:32,060 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_3 | 2019-10-15 03:16:43,723 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:32,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:43,778 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:32,060 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 755 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:32,075 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection675: begin an election at term 756 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:32,077 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,348 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 672 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:20,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:20,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 667 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,349 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,394 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 668 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,395 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,395 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:20,395 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:20,476 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:20,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,476 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 672 for changeToCandidate datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:20,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:32,103 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection675: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t756, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t756] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t756, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:43,809 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:32,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 756 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:20,499 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: begin an election at term 673 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:20,504 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:32,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:32,104 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:43,925 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 839 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:43,925 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:43,925 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:43,925 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:43,927 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,141 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:32,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:32,141 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 756 for changeToCandidate datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,515 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t673, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:32,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,515 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:32,165 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection676: begin an election at term 757 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,193 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection676: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t757, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t757] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t757, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:32,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 757 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:32,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:32,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:32,226 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:43,940 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 840 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:43,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:43,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:43,940 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:44,137 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 841 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:44,137 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:44,137 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:44,137 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:44,138 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | ... 1 more datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,519 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: begin an election at term 674 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:20,523 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:32,289 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:44,188 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:32,315 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,525 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t674, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,526 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:44,225 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:32,320 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:32,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:32,320 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 757 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:32,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:32,327 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection677: begin an election at term 758 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:32,366 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection677: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t758] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t758, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:32,391 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection677: begin an election at term 759 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:32,392 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:44,241 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | ... 1 more datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:20,549 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: begin an election at term 675 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,551 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 669 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:20,551 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:20,554 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:32,419 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection677: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t759, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t759] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t759, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:32,419 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 759 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:32,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:32,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:32,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_3 | 2019-10-15 03:16:44,262 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:32,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 759 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:32,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:32,539 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection678: begin an election at term 760 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:32,540 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:20,574 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t675, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,574 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:44,339 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 842 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:32,555 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection678: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t760, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t760] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t760, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:44,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:32,555 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 760 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:44,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:32,555 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:44,339 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:32,555 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:44,403 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 843 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:32,645 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:44,403 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:32,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:44,403 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,645 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 760 for changeToCandidate datanode_3 | 2019-10-15 03:16:44,403 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:32,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:44,482 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,659 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection679: begin an election at term 761 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:32,661 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,597 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: begin an election at term 676 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:44,496 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:20,602 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603 got exception when requesting votes: {} datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:32,687 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection679: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t761, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t761] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t761, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:32,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 761 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:32,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:32,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:32,757 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:32,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:32,757 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 761 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | 2019-10-15 03:16:32,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:32,771 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection680: begin an election at term 762 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:32,787 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection680: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t762, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t762] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t762, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:32,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 762 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:32,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:32,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:32,807 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:44,527 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 844 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:44,527 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:44,528 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:44,528 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:44,573 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | 2019-10-15 03:16:20,613 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection603: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t676] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t676, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:20,613 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:32,951 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:32,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | 2019-10-15 03:16:32,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 762 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:32,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:32,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection681: begin an election at term 763 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:32,980 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:44,591 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | ... 1 more datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,613 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 676 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,613 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,613 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,650 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 670 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,650 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:20,736 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 671 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:20,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:20,736 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:32,986 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection681: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t763, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,736 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:32,994 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection681: begin an election at term 764 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,756 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:16:44,718 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:33,022 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection681: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t764, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t764] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t764, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,756 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:33,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 764 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:20,756 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 676 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:33,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:20,756 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:33,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:20,760 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection604: begin an election at term 677 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:33,130 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,766 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection604 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:33,130 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_1 | 2019-10-15 03:16:33,130 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 764 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | 2019-10-15 03:16:33,130 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | 2019-10-15 03:16:33,148 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection682: begin an election at term 765 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:33,150 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:44,737 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 845 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:44,737 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:44,737 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:44,737 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:44,815 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 846 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:44,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:44,815 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:44,815 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:33,164 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection682: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t765, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t765] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t765, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:44,907 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | 2019-10-15 03:16:33,164 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 765 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:33,164 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:20,769 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection604: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t677] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t677, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | 2019-10-15 03:16:33,164 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:20,769 INFO impl.LeaderElection: Exception 0: {} datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | 2019-10-15 03:16:33,166 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | 2019-10-15 03:16:33,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:33,166 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 765 for changeToCandidate datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:33,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:33,182 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection683: begin an election at term 766 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:33,208 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection683: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t766, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t766] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t766, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:33,208 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 766 for DISCOVERED_A_NEW_TERM datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | 2019-10-15 03:16:33,208 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:33,208 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:33,222 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:45,035 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 847 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:45,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:45,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:45,035 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:45,118 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:20,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 677 for DISCOVERED_A_NEW_TERM datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:20,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:33,360 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,789 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_1 | 2019-10-15 03:16:33,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:33,360 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 766 for changeToCandidate datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 677 for changeToCandidate datanode_1 | 2019-10-15 03:16:33,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,812 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection605: begin an election at term 678 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | 2019-10-15 03:16:33,376 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection684: begin an election at term 767 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,817 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection605 got exception when requesting votes: {} datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | 2019-10-15 03:16:33,405 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection684: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t767, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t767] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t767, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | 2019-10-15 03:16:33,405 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 767 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:33,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:33,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:33,436 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:45,259 WARN server.GrpcServerProtocolService: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | org.apache.ratis.protocol.GroupMismatchException: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:45,260 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 848 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:33,601 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | 2019-10-15 03:16:45,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:33,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:45,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:33,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 767 for changeToCandidate datanode_3 | 2019-10-15 03:16:45,260 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:33,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:20,829 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection605: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t678] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t678, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:49,819 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 849 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection685: begin an election at term 768 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:20,829 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:49,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:33,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection685: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t768, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t768] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t768, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:49,820 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:33,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 768 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:49,820 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:33,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:49,890 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 850 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:33,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:49,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:33,650 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:49,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:49,890 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:50,103 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 851 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:50,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:50,104 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:50,104 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:50,159 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 852 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:50,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:50,160 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:50,160 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:50,235 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 853 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:50,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:50,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:50,235 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:50,250 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 854 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:50,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:33,690 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:20,829 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 678 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:50,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:20,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:50,250 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:20,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:50,277 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 855 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:20,869 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 672 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:50,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:20,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:20,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:50,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:20,869 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:50,277 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:20,941 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_3 | 2019-10-15 03:16:50,304 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 856 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:20,941 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:50,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:20,941 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 678 for changeToCandidate datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,941 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,966 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection606: begin an election at term 679 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,971 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection606 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:50,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:50,305 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:50,460 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 857 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:50,460 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:50,461 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:33,774 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:16:50,461 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:50,569 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 858 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:50,569 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:33,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 768 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:50,569 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:33,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:50,569 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:33,789 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection686: begin an election at term 769 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:50,658 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 859 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,790 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:50,658 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:50,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:50,659 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:50,684 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 860 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:50,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:50,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:50,684 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:50,699 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 861 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:50,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:50,699 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:20,983 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection606: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t679] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t679, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:50,699 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:20,983 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:50,803 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 862 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:50,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:50,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:50,803 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:50,975 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 863 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,817 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection686: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t769, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t769] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t769, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:50,975 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:16:50,976 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:33,817 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 769 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:50,976 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:33,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:51,008 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 864 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:33,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:51,008 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:33,854 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_3 | 2019-10-15 03:16:51,008 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:33,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:51,008 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:33,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 769 for changeToCandidate datanode_3 | 2019-10-15 03:16:51,267 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 865 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:33,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:33,858 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:51,267 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:51,268 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:51,268 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:51,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 866 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:51,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:51,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:51,362 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:20,983 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 679 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:51,589 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 867 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:20,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:51,589 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:20,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:51,589 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:21,018 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 673 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:51,589 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:21,018 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:51,772 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 868 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:21,018 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:51,772 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,018 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:51,772 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,120 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | 2019-10-15 03:16:51,772 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:21,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:51,959 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 869 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,120 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 679 for changeToCandidate datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:51,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection687: begin an election at term 770 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,135 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection607: begin an election at term 680 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:51,959 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:33,875 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection687: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t770, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t770] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t770, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,138 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 674 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:51,960 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:33,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 770 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:21,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:52,077 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 870 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:21,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:52,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:33,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:21,138 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:52,078 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:33,917 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:21,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection607 got exception when requesting votes: {} datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:52,078 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:52,188 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 871 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:52,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:52,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:52,188 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:52,290 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 872 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:52,290 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:52,290 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:52,290 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:52,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 873 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:52,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:52,442 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:52,442 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:52,640 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 874 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:52,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:52,641 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:52,641 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:33,943 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:52,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 875 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:52,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:33,943 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 770 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:52,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:33,943 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:52,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:33,956 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection688: begin an election at term 771 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:52,911 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 876 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:33,993 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection688: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t771, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t771] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t771, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:52,911 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,164 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection607: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t680] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t680, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:33,993 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 771 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:52,911 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,164 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:33,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:52,911 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:33,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:53,144 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 877 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:34,059 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:53,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:53,144 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:53,144 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:53,216 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 878 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:53,216 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:53,217 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:53,217 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:53,461 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 879 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:53,461 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:53,461 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:53,461 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:53,648 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 880 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:53,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:53,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:53,648 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:53,790 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 881 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:53,790 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:53,790 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,125 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:53,790 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,165 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 680 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:54,031 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 882 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:34,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:54,031 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,165 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:34,126 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 771 for changeToCandidate datanode_3 | 2019-10-15 03:16:54,031 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,187 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 675 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:34,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:54,031 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:34,151 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection689: begin an election at term 772 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,114 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: addNew group-357F199BEF59:[cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-357F199BEF59:java.util.concurrent.CompletableFuture@10cc4896[Not completed] datanode_2 | 2019-10-15 03:16:21,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:34,179 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection689: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t772, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t772] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t772, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,120 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: new RaftServerImpl for group-357F199BEF59:[cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized datanode_2 | 2019-10-15 03:16:21,187 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,179 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 772 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:54,120 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_2 | 2019-10-15 03:16:21,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_1 | 2019-10-15 03:16:34,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:54,121 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_2 | 2019-10-15 03:16:21,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:34,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:54,121 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_2 | 2019-10-15 03:16:21,230 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 680 for changeToCandidate datanode_3 | 2019-10-15 03:16:54,121 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_1 | 2019-10-15 03:16:34,191 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | 2019-10-15 03:16:21,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:54,121 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | 2019-10-15 03:16:34,192 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,234 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection608: begin an election at term 681 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,121 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: ConfigurationManager, init=-1: [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_1 | 2019-10-15 03:16:34,192 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 772 for changeToCandidate datanode_3 | 2019-10-15 03:16:54,121 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_1 | 2019-10-15 03:16:34,192 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:54,122 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | 2019-10-15 03:16:21,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection608 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:34,218 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection690: begin an election at term 773 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,122 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/1c05e746-3997-432d-a303-357f199bef59 does not exist. Creating ... datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:34,220 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:54,139 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/1c05e746-3997-432d-a303-357f199bef59/in_use.lock acquired by nodename 7@074e53f4bc48 datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,144 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/1c05e746-3997-432d-a303-357f199bef59 has been successfully formatted. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:54,144 [pool-8-thread-1] INFO - group-357F199BEF59: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,145 INFO segmented.SegmentedRaftLogWorker: new cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/1c05e746-3997-432d-a303-357f199bef59 datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:54,145 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:54,146 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | 2019-10-15 03:16:34,229 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection690: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t773, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t773] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t773, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:54,147 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_1 | 2019-10-15 03:16:34,229 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 773 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:54,147 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_1 | 2019-10-15 03:16:34,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:54,147 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_1 | 2019-10-15 03:16:34,230 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:21,240 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection608: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t681] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t681, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,147 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | 2019-10-15 03:16:21,240 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:34,343 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_3 | 2019-10-15 03:16:54,148 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:34,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:54,154 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: start as a follower, conf=-1: [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:34,343 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 773 for changeToCandidate datanode_3 | 2019-10-15 03:16:54,154 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:54,154 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:34,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:34,357 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection691: begin an election at term 774 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,155 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-357F199BEF59,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:34,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection691: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t774, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t774] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t774, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,272 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 883 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:34,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 774 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:54,272 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,272 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:34,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:54,272 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:54,285 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_1 | 2019-10-15 03:16:34,433 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,286 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:54,286 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:54,286 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:54,301 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-LeaderElection4: begin an election at term 1 for -1: [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:54,301 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:54,301 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:54,301 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: change Leader from null to cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at term 1 for becomeLeader, leader elected after 156ms datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:54,301 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:54,301 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:54,302 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:21,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 681 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:54,302 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:21,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:54,302 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:21,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:54,302 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:21,371 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 676 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:54,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:21,371 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:34,451 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_3 | 2019-10-15 03:16:54,305 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | 2019-10-15 03:16:21,372 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:34,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:54,306 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59: set configuration 0: [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 datanode_2 | 2019-10-15 03:16:21,372 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,451 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 774 for changeToCandidate datanode_3 | 2019-10-15 03:16:54,329 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 884 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,439 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_1 | 2019-10-15 03:16:34,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:21,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:34,465 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: begin an election at term 775 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,329 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,439 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 681 for changeToCandidate datanode_1 | 2019-10-15 03:16:34,493 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t775, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,329 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,439 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:34,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: begin an election at term 776 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,453 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection609: begin an election at term 682 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,329 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,550 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:21,458 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection609 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:54,356 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-357F199BEF59-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/1c05e746-3997-432d-a303-357f199bef59/current/log_inprogress_0 datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,438 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 885 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:54,438 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:54,438 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:54,438 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:54,453 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 886 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:54,453 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:54,453 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:54,453 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:54,482 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 887 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:54,482 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:54,483 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:54,483 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:54,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 888 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:54,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:54,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:54,611 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:34,570 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t776, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,703 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 889 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:34,583 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: begin an election at term 777 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,703 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:34,598 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection692: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t777, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t777] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t777, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:54,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:34,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 777 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:54,704 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:34,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:54,754 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 890 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:34,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:54,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:34,676 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:54,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,470 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection609: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t682] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t682, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:21,470 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:54,754 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:54,999 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 891 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:54,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:54,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:54,999 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:55,119 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 892 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:55,119 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:55,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:55,120 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:55,202 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 893 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:55,202 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:55,202 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:55,202 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:55,247 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 894 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:55,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:55,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:34,700 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_3 | 2019-10-15 03:16:55,247 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:34,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:55,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 895 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:34,700 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 777 for changeToCandidate datanode_3 | 2019-10-15 03:16:55,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:34,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:55,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:34,715 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection693: begin an election at term 778 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:55,362 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:55,409 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 896 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:34,731 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection693: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t778, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:55,409 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:34,736 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection693: begin an election at term 779 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:16:55,409 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:34,773 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection693: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t779, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t779] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t779, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,470 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 682 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:34,773 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 779 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:21,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:55,409 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:21,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:55,544 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 897 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:34,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:55,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 677 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:34,796 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:21,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:55,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:21,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:55,544 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:21,484 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:55,696 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 898 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:21,599 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_3 | 2019-10-15 03:16:55,696 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:21,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:55,696 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:21,599 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 682 for changeToCandidate datanode_3 | 2019-10-15 03:16:55,696 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:21,599 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:55,734 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 899 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:21,613 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection610: begin an election at term 683 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:55,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,617 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 678 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:55,734 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,617 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:55,734 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,617 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:55,926 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 900 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,617 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:55,926 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,620 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection610 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:55,926 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:55,926 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:56,141 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 901 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:34,888 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_3 | 2019-10-15 03:16:56,141 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:34,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:56,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:34,888 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 779 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:56,142 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:34,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:56,330 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 902 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:34,913 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection694: begin an election at term 780 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:56,330 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:34,941 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection694: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t780, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t780] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t780, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:56,330 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:16:56,330 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:34,941 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 780 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:34,941 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:56,519 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 903 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:34,941 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:56,519 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,008 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:56,519 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:56,519 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:56,673 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 904 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:56,674 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:56,674 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:56,674 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:21,630 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection610: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t683] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t683, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:56,704 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 905 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:21,630 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:16:56,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:56,704 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:56,704 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:56,745 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 906 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:56,745 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:56,745 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:56,745 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:56,934 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 907 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:56,934 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:56,934 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:35,138 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | 2019-10-15 03:16:56,935 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:35,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:16:57,001 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 908 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:35,138 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 780 for changeToCandidate datanode_3 | 2019-10-15 03:16:57,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:35,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:57,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:35,158 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection695: begin an election at term 781 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:57,002 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,159 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:57,071 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 909 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:16:57,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:57,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:57,071 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:57,282 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 910 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:57,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:57,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:57,282 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:57,359 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 911 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:57,359 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:16:57,359 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:57,359 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:57,567 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 912 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:57,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,630 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 683 for DISCOVERED_A_NEW_TERM datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:57,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,630 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:57,568 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,630 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:57,585 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 913 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,721 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_1 | 2019-10-15 03:16:35,184 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection695: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t781, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t781] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t781, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:57,585 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 781 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:57,586 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,721 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 683 for changeToCandidate datanode_1 | 2019-10-15 03:16:35,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:21,721 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:21,735 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection611: begin an election at term 684 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:35,185 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:16:57,586 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,740 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection611 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:35,185 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:57,749 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 914 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:57,750 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:57,750 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:57,750 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:57,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 915 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:57,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:57,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:57,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:58,112 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 916 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:58,112 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:16:58,113 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:16:58,113 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:16:58,277 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 917 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:16:58,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:16:58,277 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:16:58,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:16:58,472 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 918 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:58,472 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,247 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:16:58,472 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:16:58,472 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:58,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 919 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:58,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:58,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:58,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,751 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection611: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t684] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t684, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:58,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 920 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,751 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:58,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:58,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:58,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:58,752 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 921 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:58,752 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:58,752 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:58,752 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:58,809 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 922 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:58,809 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:16:58,809 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:16:58,809 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:35,363 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:16:58,870 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 923 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:58,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:16:58,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:16:58,870 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:16:58,969 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 924 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:16:58,969 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:16:58,969 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:16:58,969 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:16:59,096 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 925 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:16:59,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,751 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 684 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:16:59,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:21,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:16:59,097 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:21,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:21,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 679 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:16:59,306 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 926 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:16:59,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:59,306 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:21,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:16:59,306 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:21,770 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:16:59,498 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 927 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,382 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | 2019-10-15 03:16:21,889 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_3 | 2019-10-15 03:16:59,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:16:59,498 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,382 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 781 for changeToCandidate datanode_3 | 2019-10-15 03:16:59,498 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,382 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:21,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 684 for changeToCandidate datanode_3 | 2019-10-15 03:16:59,569 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 928 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection696: begin an election at term 782 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:16:59,569 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,411 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection696: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t782, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t782] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t782, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,913 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection612: begin an election at term 685 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:59,569 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,411 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 782 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:21,917 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection612 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:16:59,569 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:16:59,795 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 929 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:16:59,795 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,457 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:16:59,796 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,457 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:16:59,796 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 782 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:16:59,962 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 930 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:16:59,962 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:35,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:16:59,962 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:35,471 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection697: begin an election at term 783 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:16:59,962 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:35,498 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection697: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t783, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t783] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t783, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:00,122 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 931 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:35,498 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 783 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:00,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:35,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:00,122 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:00,122 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,556 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:00,273 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 932 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:00,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:00,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:00,273 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:00,352 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 933 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:00,352 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:00,353 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:00,353 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:00,449 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 934 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:00,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,939 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection612: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t685] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t685, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:00,449 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,939 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:00,449 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:00,553 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 935 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:00,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:00,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:00,553 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:00,726 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 936 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:35,681 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_3 | 2019-10-15 03:17:00,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:35,681 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:00,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:35,681 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 783 for changeToCandidate datanode_3 | 2019-10-15 03:17:00,726 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:00,870 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 937 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,681 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:00,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,706 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection698: begin an election at term 784 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:00,870 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,731 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection698: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t784, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:00,870 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,746 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection698: begin an election at term 785 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:35,747 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:01,108 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 938 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:01,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:01,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:01,108 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:01,257 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 939 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:01,257 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:01,257 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:21,939 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 685 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:01,257 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:21,939 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:01,399 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 940 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:01,399 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,939 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:01,399 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,940 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 680 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:01,399 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:21,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:01,477 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 941 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:01,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,940 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:01,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,968 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:01,477 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:01,550 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 942 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:01,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:21,968 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 685 for changeToCandidate datanode_3 | 2019-10-15 03:17:01,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:01,550 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,761 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection698: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t785, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t785] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t785, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:21,981 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection613: begin an election at term 686 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:01,577 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 943 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,986 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection613 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:35,761 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 785 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:01,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:01,577 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:01,577 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:35,894 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:01,763 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 944 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:01,764 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:01,764 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:01,764 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:01,792 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 945 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:01,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:01,793 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:01,793 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:01,819 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 946 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:01,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:01,819 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:01,819 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:02,040 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 947 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:02,040 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:02,040 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:02,040 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:02,138 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 948 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,916 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:02,138 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:02,138 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:21,997 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection613: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t686] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t686, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:02,139 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:21,997 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:02,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 949 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:02,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:02,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:02,270 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:02,428 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 950 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:02,428 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:02,428 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:02,428 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:02,467 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 951 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:02,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:02,467 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:02,467 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:02,589 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 952 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:35,959 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_3 | 2019-10-15 03:17:02,589 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:35,959 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:02,589 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:35,959 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 785 for changeToCandidate datanode_3 | 2019-10-15 03:17:02,589 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:02,794 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 953 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,959 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:02,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,983 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection699: begin an election at term 786 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:02,794 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:35,998 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection699: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t786, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t786] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t786, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:02,795 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:35,998 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 786 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:03,001 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 954 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:35,998 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:03,001 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:35,998 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:03,001 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,077 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:21,997 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 686 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:03,001 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:21,997 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:03,178 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 955 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:21,998 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:03,178 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,067 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:03,178 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,067 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:22,067 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 686 for changeToCandidate datanode_3 | 2019-10-15 03:17:03,178 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:22,067 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:03,410 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 956 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,082 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection614: begin an election at term 687 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:03,410 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,086 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection614 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:03,410 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:03,410 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:03,623 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 957 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:03,623 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:03,623 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:03,623 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:03,850 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 958 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:03,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:36,160 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_3 | 2019-10-15 03:17:03,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:36,160 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:36,160 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 786 for changeToCandidate datanode_3 | 2019-10-15 03:17:03,850 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:36,160 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:03,878 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 959 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:36,172 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection700: begin an election at term 787 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:03,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:36,178 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection700: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t787, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t787] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t787, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:03,878 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:36,179 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 787 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:03,878 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:36,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:04,016 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 960 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,280 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:04,016 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:04,016 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:04,016 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:04,117 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 961 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,109 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection614: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t687] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t687, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:04,117 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,109 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:04,117 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:04,117 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:04,288 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 962 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:04,288 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:04,288 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:04,288 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:04,383 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 963 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:04,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:04,383 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:04,383 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:04,419 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 964 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,330 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:04,419 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:04,419 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,330 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 787 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:04,419 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:36,344 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection701: begin an election at term 788 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:04,497 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 965 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,373 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection701: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t788, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t788] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t788, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:04,497 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,373 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 788 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:04,497 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:04,497 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,373 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:04,649 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 966 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,480 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:04,649 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:04,649 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 687 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:04,650 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:04,741 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 967 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:04,741 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 681 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:04,741 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:04,741 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:04,859 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 968 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,110 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:04,859 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,174 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 682 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:04,859 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:04,859 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:22,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:04,986 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 969 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:22,174 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:04,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,228 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 683 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:04,986 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:04,986 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:05,062 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 970 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,228 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:05,062 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,272 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_1 | 2019-10-15 03:16:36,527 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_3 | 2019-10-15 03:17:05,062 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,273 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 687 for changeToCandidate datanode_3 | 2019-10-15 03:17:05,062 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:36,528 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 788 for changeToCandidate datanode_3 | 2019-10-15 03:17:05,101 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 971 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,297 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection615: begin an election at term 688 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:36,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:05,102 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection615 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:36,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection702: begin an election at term 789 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:05,102 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:36,574 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection702: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t789, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:05,102 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,591 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection702: begin an election at term 790 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:05,226 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 972 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,628 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection702: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t790, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t790] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t790, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:05,226 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 790 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:05,226 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:36,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:05,226 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:36,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:05,263 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 973 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:05,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,665 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:05,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:05,263 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,665 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 790 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:05,394 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 974 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,665 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:05,394 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,679 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection703: begin an election at term 791 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:05,394 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,707 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection703: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t791, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t791] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t791, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:05,394 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:36,707 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 791 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:05,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 975 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:05,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,710 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:05,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:05,611 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:05,799 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 976 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:05,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:05,799 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:05,799 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,313 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection615: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t688] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t688, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:05,813 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 977 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,313 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:05,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:05,813 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:05,813 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:05,953 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 978 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:05,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:05,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:05,953 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:05,982 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 979 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:05,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:05,982 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:05,982 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:06,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 980 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:36,788 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:06,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:06,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,788 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 791 for changeToCandidate datanode_3 | 2019-10-15 03:17:06,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:36,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:06,188 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 981 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:36,802 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection704: begin an election at term 792 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:06,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,825 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection704: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t792, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t792] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t792, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:06,189 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:36,825 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 792 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:06,189 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:36,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:06,291 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 982 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:06,291 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:06,291 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:36,854 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:06,291 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:22,313 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 688 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:06,532 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 983 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:06,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:06,532 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,421 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_3 | 2019-10-15 03:17:06,532 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:22,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:06,628 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 984 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:22,421 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 688 for changeToCandidate datanode_3 | 2019-10-15 03:17:06,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:22,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:06,628 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:22,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection616: begin an election at term 689 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:06,628 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:06,747 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 985 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection616 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:06,747 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:06,747 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:06,747 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:06,929 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 986 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:06,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:06,929 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:06,929 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:07,150 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 987 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:07,151 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:36,982 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:07,151 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:07,151 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:07,307 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 988 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:07,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:07,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:07,307 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:07,556 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 989 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:07,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:07,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:07,556 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:07,614 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 990 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,463 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection616: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t689] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t689, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:07,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,463 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:07,614 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:07,614 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:07,694 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 991 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:07,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:07,695 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:36,995 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:07,695 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:07,731 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 992 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:07,731 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:07,731 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:07,731 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:07,780 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 993 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:07,780 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:07,780 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:07,781 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:07,949 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 994 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:07,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:07,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:07,949 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:08,002 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 995 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:08,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:08,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:08,002 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:08,083 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 996 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:08,083 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:22,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 689 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:08,083 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,000 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | 2019-10-15 03:16:22,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:08,083 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 684 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:08,179 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 997 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,000 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 792 for changeToCandidate datanode_2 | 2019-10-15 03:16:22,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:08,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:22,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:08,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection705: begin an election at term 793 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:22,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:08,179 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,042 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection705: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t793, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t793] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t793, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:22,464 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:08,331 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 998 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,042 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 793 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:22,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_3 | 2019-10-15 03:17:08,331 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:22,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:08,331 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:22,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 689 for changeToCandidate datanode_3 | 2019-10-15 03:17:08,331 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,189 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:22,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:08,402 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 999 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:22,512 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection617: begin an election at term 690 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:08,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,516 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection617 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:08,402 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:08,402 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:08,648 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1000 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:08,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:08,648 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:08,648 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:08,861 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1001 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:08,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:08,861 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:08,861 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:08,979 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1002 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:08,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:08,979 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:37,222 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_3 | 2019-10-15 03:17:08,979 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:37,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:37,222 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 793 for changeToCandidate datanode_3 | 2019-10-15 03:17:09,070 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1003 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:37,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:09,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:37,237 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection706: begin an election at term 794 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:09,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,267 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection706: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t794] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t794, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:09,070 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:37,291 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection706: begin an election at term 795 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:09,208 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1004 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,292 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:09,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:09,208 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:09,208 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:09,298 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1005 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:09,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,528 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection617: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t690] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t690, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:09,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,528 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:09,298 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:09,398 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1006 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:09,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:09,398 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:09,398 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:09,490 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1007 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:09,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:09,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:09,490 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:09,697 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1008 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:09,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:09,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:09,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:09,728 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1009 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,308 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection706: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t795, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t795] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t795, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:09,728 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,308 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 795 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:09,728 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:09,728 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,308 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:09,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1010 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,447 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:138ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:09,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,447 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:09,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,447 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 795 for changeToCandidate datanode_2 | 2019-10-15 03:16:22,528 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 690 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:37,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:22,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:09,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,450 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:22,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:10,002 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1011 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:22,691 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_3 | 2019-10-15 03:17:10,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:22,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:10,002 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:22,691 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 690 for changeToCandidate datanode_3 | 2019-10-15 03:17:10,002 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:22,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:10,070 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1012 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:22,693 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 685 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:10,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:22,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:10,070 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:10,070 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,693 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:10,129 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1013 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,695 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection618: begin an election at term 691 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:10,129 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,699 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection618 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:10,129 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:10,129 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:10,380 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1014 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:10,380 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:10,380 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:10,381 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:10,413 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1015 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:37,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection707: begin an election at term 796 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:10,413 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,463 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:10,413 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:10,413 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:10,595 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1016 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:10,595 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:10,595 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:10,595 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:10,686 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1017 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:10,686 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:10,686 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:10,686 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:10,769 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1018 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:10,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:10,769 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:10,769 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:10,990 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1019 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,701 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection618: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t691] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t691, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:10,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:10,990 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:22,701 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:10,990 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:11,075 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1020 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:37,492 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection707: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t796, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t796] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t796, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:11,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:37,492 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 796 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:11,075 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:37,492 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:11,076 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:37,492 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:11,272 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1021 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_3 | 2019-10-15 03:17:11,272 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:37,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:11,272 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:37,512 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 796 for changeToCandidate datanode_3 | 2019-10-15 03:17:11,272 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:11,290 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1022 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:11,290 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,526 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection708: begin an election at term 797 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:11,290 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,542 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:11,290 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:11,464 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1023 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:11,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:11,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:11,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:11,601 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1024 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:11,601 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:11,601 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:11,601 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:11,655 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1025 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:11,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:22,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 691 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:11,655 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:11,655 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:11,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1026 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:22,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:11,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,759 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 686 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:11,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,759 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:11,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,545 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection708: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t797, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t797] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t797, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:22,759 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:11,931 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1027 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,545 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 797 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:22,759 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:11,931 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,545 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:22,893 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_3 | 2019-10-15 03:17:11,931 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,545 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:22,893 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:11,931 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,604 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | 2019-10-15 03:16:22,893 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 691 for changeToCandidate datanode_3 | 2019-10-15 03:17:11,946 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1028 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:22,893 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:11,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 797 for changeToCandidate datanode_2 | 2019-10-15 03:16:22,907 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection619: begin an election at term 692 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:11,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:22,912 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection619 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:11,946 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:37,617 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection709: begin an election at term 798 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:12,082 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1029 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:37,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection709: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t798, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t798] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t798, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:12,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:37,657 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 798 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:12,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:37,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:12,082 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:37,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:12,111 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1030 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:37,684 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:12,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:12,111 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:12,111 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:12,277 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1031 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:12,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:12,277 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:12,277 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:12,328 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1032 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:12,328 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:12,328 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:12,328 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:12,490 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1033 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:12,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:12,490 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:12,490 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:22,924 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection619: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t692] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t692, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:12,659 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1034 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:22,924 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:12,660 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:12,660 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,714 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:12,660 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:12,853 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1035 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:12,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:37,714 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:12,853 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,714 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 798 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:12,853 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:13,049 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1036 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,728 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection710: begin an election at term 799 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:13,049 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,762 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection710: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t799, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t799] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t799, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:13,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,762 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 799 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:13,050 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:37,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:13,077 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1037 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,797 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:13,077 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:13,077 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,797 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 799 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:13,077 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:13,209 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1038 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,813 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection711: begin an election at term 800 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:13,209 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,814 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:13,209 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:13,209 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:13,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1039 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:13,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:13,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:13,371 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:13,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1040 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:22,924 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 692 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:22,924 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:13,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:22,924 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:13,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:22,959 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 687 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:13,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:13,516 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1041 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:22,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:13,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:22,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:13,516 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:22,959 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:13,516 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 688 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:13,597 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1042 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:37,840 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection711: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t800, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t800] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t800, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:13,597 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:37,840 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 800 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:13,597 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,049 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:13,597 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,061 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_1 | 2019-10-15 03:16:37,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:13,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1043 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:37,884 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_2 | 2019-10-15 03:16:23,061 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:13,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,061 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 692 for changeToCandidate datanode_1 | 2019-10-15 03:16:37,884 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:13,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,061 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:37,885 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 800 for changeToCandidate datanode_3 | 2019-10-15 03:17:13,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,075 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection620: begin an election at term 693 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:37,885 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:13,821 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1044 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,080 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection620 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:37,899 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection712: begin an election at term 801 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:13,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:13,821 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:37,921 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection712: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t801, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:13,821 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:37,934 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection712: begin an election at term 802 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:13,926 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1045 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:13,926 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:13,926 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:13,926 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:14,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1046 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:14,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:14,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:37,952 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection712: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t802, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t802] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t802, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:14,041 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:37,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 802 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:14,256 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1047 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:37,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:14,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:37,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:14,256 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:38,018 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:14,256 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:14,347 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1048 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:14,347 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:14,347 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:14,347 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:14,555 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1049 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:14,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:14,555 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:23,092 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection620: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t693] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t693, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:14,555 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:14,583 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1050 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:14,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:14,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:14,583 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:14,701 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1051 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:14,701 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,092 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:14,701 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:14,701 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:14,888 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1052 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:14,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:14,888 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:38,032 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_3 | 2019-10-15 03:17:14,889 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:38,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:15,095 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1053 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:15,095 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 802 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:38,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:15,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,048 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection713: begin an election at term 803 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:15,096 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:38,049 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:15,196 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1054 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:15,197 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:15,197 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:15,197 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:15,300 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1055 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:15,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:15,300 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:15,300 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:15,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1056 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:15,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:15,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:15,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:15,711 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1057 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:15,711 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:15,712 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,092 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 693 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:15,712 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:15,879 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1058 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:15,879 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 689 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:15,879 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,063 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection713: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t803, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t803] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t803, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:15,879 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:38,063 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 803 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:15,955 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1059 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,179 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:23,179 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:17:15,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:15,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:17:15,955 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,179 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 693 for changeToCandidate datanode_1 | 2019-10-15 03:16:38,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:16,048 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1060 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:38,119 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:16,048 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,192 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection621: begin an election at term 694 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:16,048 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,197 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection621 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:16,048 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:16,279 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1061 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:16,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:16,279 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:16,279 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:16,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1062 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:16,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:16,471 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:16,471 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:16,653 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1063 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:16,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:16,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:16,653 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:16,848 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1064 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:16,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:16,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,149 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:85ms, electionTimeout:85ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:16,848 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:17,060 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1065 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:17,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,149 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 803 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:17,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:17,060 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,154 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection714: begin an election at term 804 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:23,198 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection621: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t694] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t694, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:17,158 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1066 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,184 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection714: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t804, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t804] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t804, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:23,198 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:17,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:17,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,184 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 804 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:17,158 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:17,350 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1067 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,184 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:17,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,263 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:17,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:17,350 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 804 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:17,376 [Command processor thread] ERROR - Container #4 does not exist in datanode. Container close failed. datanode_1 | 2019-10-15 03:16:38,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:17,377 [Command processor thread] ERROR - Container #5 does not exist in datanode. Container close failed. datanode_1 | 2019-10-15 03:16:38,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection715: begin an election at term 805 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:17,567 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1068 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,307 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection715: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t805, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t805] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t805, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:17,567 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,307 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 805 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:17,567 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,307 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:17,567 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,307 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:17,591 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1069 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,310 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:17,591 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:17,591 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:17,591 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:17,687 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1070 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:17,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:17,687 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:17,688 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:17,738 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1071 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,198 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 694 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:17,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,198 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:17,738 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,198 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:17,738 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,311 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:17,805 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1072 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:17,805 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:23,311 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 694 for changeToCandidate datanode_3 | 2019-10-15 03:17:17,805 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:23,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:17,805 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:23,325 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection622: begin an election at term 695 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:17,845 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1073 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:23,329 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection622: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t695, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:17,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:17,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,361 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection622: begin an election at term 696 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:23,365 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 690 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:17,845 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,411 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:23,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:17,875 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1074 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:17,875 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:23,365 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:17,875 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:23,368 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection622 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:17,875 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:18,026 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1075 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:18,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:18,026 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:18,026 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:18,179 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1076 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:18,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:18,180 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:18,180 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:18,323 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1077 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:18,323 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:18,323 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:18,323 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:18,415 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1078 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:18,415 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:18,415 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:18,415 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,482 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:18,553 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1079 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:18,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,482 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 805 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:18,553 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:18,553 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,505 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection716: begin an election at term 806 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:18,642 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1080 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,369 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection622: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t696] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t696, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:38,521 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection716: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t806, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t806] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t806, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:18,642 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,369 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:38,521 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 806 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:18,642 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:18,642 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,521 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:18,659 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1081 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:18,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,582 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:18,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:18,659 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:18,850 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1082 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:18,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:18,850 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:18,850 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:19,059 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1083 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:19,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:19,060 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:19,235 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1084 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:19,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:19,235 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:19,235 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:19,452 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1085 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:19,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:19,452 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,650 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:19,452 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:38,650 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 806 for changeToCandidate datanode_2 | 2019-10-15 03:16:23,369 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 696 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:19,563 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1086 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:23,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:19,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:38,665 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection717: begin an election at term 807 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:23,434 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_1 | 2019-10-15 03:16:38,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection717: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t807, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t807] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t807, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:19,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,695 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 807 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:19,563 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,434 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 696 for changeToCandidate datanode_1 | 2019-10-15 03:16:38,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:19,633 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1087 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:38,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:19,633 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,449 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection623: begin an election at term 697 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:38,698 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:19,633 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,452 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 691 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:19,633 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:19,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1088 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:19,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:23,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:19,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:23,452 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:19,774 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:23,454 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection623 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:19,881 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1089 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:19,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:19,881 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:19,881 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:20,112 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1090 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:20,112 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:20,112 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:20,112 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:20,140 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1091 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:20,140 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:20,140 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,723 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:20,140 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:20,274 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1092 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:20,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:20,275 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:20,275 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:20,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1093 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:20,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:20,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:20,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:20,602 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1094 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:20,602 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:20,602 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,476 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection623: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t697] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t697, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:20,602 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:20,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1095 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,476 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:20,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:20,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,826 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:20,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:20,802 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1096 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:20,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:20,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:20,802 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:20,960 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1097 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:20,960 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:20,960 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:20,960 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:20,987 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1098 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:20,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:20,987 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:20,987 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:21,080 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1099 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:21,080 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:21,080 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:21,080 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:21,230 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1100 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,828 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_3 | 2019-10-15 03:17:21,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:21,230 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,828 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 807 for changeToCandidate datanode_3 | 2019-10-15 03:17:21,230 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:21,258 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1101 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,828 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:23,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 697 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:21,258 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,842 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection718: begin an election at term 808 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:23,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:21,258 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,858 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection718: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t808, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t808] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t808, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:23,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:21,258 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:38,858 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 808 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:23,577 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 692 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:21,478 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1102 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:38,858 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:23,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:21,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,858 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:23,577 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:21,478 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:38,985 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_2 | 2019-10-15 03:16:23,577 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:21,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,649 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_1 | 2019-10-15 03:16:38,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:21,550 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1103 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:38,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 808 for changeToCandidate datanode_3 | 2019-10-15 03:17:21,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,649 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 697 for changeToCandidate datanode_1 | 2019-10-15 03:16:38,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:23,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:21,550 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:17:21,550 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection624: begin an election at term 698 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:39,003 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection719: begin an election at term 809 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:21,740 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1104 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 693 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:21,740 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:39,004 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:23,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:21,740 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:21,740 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,677 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:21,952 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1105 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,680 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection624 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:21,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:21,952 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:21,952 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:22,039 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1106 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:22,039 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:22,039 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:22,039 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:22,263 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1107 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:22,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:22,263 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:22,263 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:22,315 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1108 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:22,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:22,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:22,315 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:39,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection719: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t809, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t809] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t809, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:22,480 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1109 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:39,021 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 809 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:22,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:39,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:22,480 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:39,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:22,480 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:39,105 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:22,662 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1110 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:22,662 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:22,662 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:22,662 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:23,691 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection624: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t698] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t698, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:22,855 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1111 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,691 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:22,856 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:22,856 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:22,856 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:22,980 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1112 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:22,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:22,980 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:22,980 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:23,058 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1113 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:23,058 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:23,058 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:23,058 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:23,147 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1114 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:39,158 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | 2019-10-15 03:17:23,147 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:23,147 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:39,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:39,158 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 809 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:23,147 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:39,158 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:23,282 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1115 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:39,172 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection720: begin an election at term 810 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:23,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:39,180 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection720: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t810, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:23,282 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:39,199 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection720: begin an election at term 811 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:23,282 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:39,214 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection720: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t811, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t811] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t811, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:23,494 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1116 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:23,494 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:39,214 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 811 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:23,495 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:17:23,495 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:39,214 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:23,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1117 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:39,214 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:23,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:39,215 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:23,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:23,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:23,709 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1118 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:23,692 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 698 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:23,709 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:23,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:23,710 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:23,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:23,710 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:23,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_3 | 2019-10-15 03:17:23,950 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1119 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:23,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:23,950 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:23,813 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 698 for changeToCandidate datanode_3 | 2019-10-15 03:17:23,950 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:23,813 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:23,950 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,827 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection625: begin an election at term 699 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:24,060 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1120 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,831 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection625 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:24,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:24,060 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:24,060 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:24,240 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1121 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:24,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:39,337 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_3 | 2019-10-15 03:17:24,240 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:39,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:24,240 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:39,337 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 811 for changeToCandidate datanode_3 | 2019-10-15 03:17:24,464 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1122 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:39,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:24,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:40,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection721: begin an election at term 812 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:24,464 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:40,026 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:24,464 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:24,692 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1123 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:24,692 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:24,692 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:24,692 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:24,864 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1124 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:24,864 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:24,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:24,865 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:24,904 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1125 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:24,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,842 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection625: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t699] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t699, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:23,842 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:24,904 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:24,904 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:24,949 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1126 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:40,052 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection721: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t812, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t812] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t812, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:24,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:40,052 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 812 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:40,052 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:24,949 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,052 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:24,949 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,054 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:25,192 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1127 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:25,192 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:25,192 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:25,192 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:25,316 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1128 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:25,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:25,316 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:25,316 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:25,531 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1129 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:25,531 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:25,531 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:25,531 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:25,565 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1130 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,842 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 699 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:25,565 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,842 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:25,565 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:25,565 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,896 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 694 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:25,579 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1131 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,896 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:25,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,896 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:23,896 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:23,961 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_3 | 2019-10-15 03:17:25,579 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,058 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_3 | 2019-10-15 03:17:25,579 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:25,762 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1132 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,961 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 699 for changeToCandidate datanode_1 | 2019-10-15 03:16:40,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 812 for changeToCandidate datanode_3 | 2019-10-15 03:17:25,762 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:40,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:25,762 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 695 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,071 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection722: begin an election at term 813 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:25,762 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection722: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t813, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:25,821 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1133 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:23,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:40,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection722: begin an election at term 814 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:25,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:23,964 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,087 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:25,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:23,965 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection626: begin an election at term 700 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:25,822 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:23,970 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection626 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:26,047 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1134 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:26,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:26,047 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:26,047 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:26,143 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1135 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:26,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:26,143 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:26,143 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:26,158 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1136 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:26,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:26,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:26,158 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:26,217 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1137 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:26,217 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,117 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection722: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t814, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t814] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t814, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:26,217 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,117 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 814 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:26,217 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:26,304 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1138 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:40,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:26,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:40,152 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:26,304 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:26,304 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:26,504 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1139 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:26,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,013 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection626: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t700] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t700, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:26,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,013 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:26,504 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:26,629 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1140 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:26,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:26,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:26,629 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:26,822 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1141 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:26,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:26,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:26,822 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:26,974 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1142 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:26,974 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:26,974 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,297 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:26,974 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:27,072 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1143 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,297 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 814 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:27,072 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:27,073 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:27,073 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:27,142 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1144 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:40,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection723: begin an election at term 815 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:27,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:40,343 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection723: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t815, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t815] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t815, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:27,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:40,343 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 815 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:27,142 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:40,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:27,348 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1145 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:40,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:27,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,014 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 700 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:40,343 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:27,348 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:27,348 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:27,416 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1146 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,127 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 696 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:27,416 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:24,127 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:27,416 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:24,127 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:27,416 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:24,127 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:27,461 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1147 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:27,461 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,144 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 697 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:27,461 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:24,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:27,461 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:27,477 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1148 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,144 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:27,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:24,160 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 698 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:27,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:24,160 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:27,477 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:27,524 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1149 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:24,160 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:27,524 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:24,160 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:27,524 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:24,181 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_3 | 2019-10-15 03:17:27,524 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,427 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:24,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:27,578 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1150 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:24,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 700 for changeToCandidate datanode_3 | 2019-10-15 03:17:27,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:24,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:27,578 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:24,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection627: begin an election at term 701 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:27,578 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:24,199 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection627 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:27,667 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1151 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:27,667 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:27,667 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:27,667 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:27,877 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1152 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:27,877 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:27,877 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:27,877 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:28,021 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1153 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:28,021 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:28,021 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,480 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:137ms, electionTimeout:137ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:28,021 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,480 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:28,162 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1154 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,481 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 815 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:28,162 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,481 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:28,162 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,495 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection724: begin an election at term 816 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:28,162 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:40,511 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection724: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t816, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t816] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t816, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:28,304 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1155 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:28,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,512 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 816 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:28,305 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:28,305 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:28,504 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1156 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,532 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:24,210 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection627: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t701] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t701, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:28,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:24,210 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:28,504 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:28,504 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:28,583 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1157 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:28,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:28,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:28,584 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:28,676 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1158 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:28,676 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:28,676 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:28,676 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:28,707 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1159 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:28,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:28,707 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:28,707 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:28,852 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1160 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:28,852 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,634 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:28,852 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:40,634 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 816 for changeToCandidate datanode_3 | 2019-10-15 03:17:28,852 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:40,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:28,903 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1161 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:40,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection725: begin an election at term 817 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:28,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:40,685 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection725: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t817, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t817] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t817, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:28,903 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:40,685 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 817 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:28,903 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:40,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:29,125 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1162 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:29,125 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:24,211 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 701 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:29,125 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,686 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:24,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:29,125 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:24,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:29,221 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1163 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:24,220 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_3 | 2019-10-15 03:17:29,221 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:29,221 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:29,221 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,220 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 701 for changeToCandidate datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:29,484 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1164 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:29,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:29,484 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:29,484 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,234 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection628: begin an election at term 702 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:29,499 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1165 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:24,238 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection628 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:29,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:29,499 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:29,499 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:29,566 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1166 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:29,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:29,566 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:29,566 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:40,826 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:29,678 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1167 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:29,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:29,678 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:29,678 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:29,890 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1168 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:29,891 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:29,891 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:29,891 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:30,041 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1169 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:30,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:30,041 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:30,041 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:30,179 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1170 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,250 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection628: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t702] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t702, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:30,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,250 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:30,179 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:30,179 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:30,251 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1171 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,844 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:30,251 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:30,251 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:30,252 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,844 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 817 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:30,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1172 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:40,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:30,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:40,859 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection726: begin an election at term 818 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:30,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:40,887 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection726: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t818, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t818] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t818, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:30,370 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:40,887 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 818 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:40,887 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:30,508 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1173 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:40,887 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:30,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:40,996 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:30,508 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:30,508 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:30,576 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1174 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:30,576 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:30,576 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:30,576 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:30,696 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1175 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:30,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:30,697 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:30,697 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 702 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:30,803 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1176 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:24,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:30,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:24,252 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_3 | 2019-10-15 03:17:30,803 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:24,252 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:30,803 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,252 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 702 for changeToCandidate datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:30,993 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1177 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,252 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:24,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection629: begin an election at term 703 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:41,006 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_3 | 2019-10-15 03:17:30,993 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,265 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection629 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:41,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:30,993 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:41,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 818 for changeToCandidate datanode_3 | 2019-10-15 03:17:30,993 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:41,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:31,053 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1178 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:41,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection727: begin an election at term 819 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:41,037 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection727: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t819, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t819] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t819, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:31,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:41,037 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 819 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:31,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:31,053 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:41,060 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:31,120 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1179 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:31,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:31,120 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:31,120 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:31,263 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1180 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:31,264 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:31,264 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:31,264 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:31,278 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1181 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:31,278 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:31,278 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:31,278 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:31,306 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1182 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:31,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:31,307 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:24,276 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection629: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t703] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t703, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:31,307 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,276 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:31,424 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1183 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:31,424 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:41,070 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:31,425 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:31,425 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:31,492 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1184 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:31,492 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:31,492 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:31,492 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:31,684 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1185 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:31,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:31,684 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:31,684 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:31,840 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1186 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:31,840 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:31,840 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:31,840 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:41,086 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_3 | 2019-10-15 03:17:31,991 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1187 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:41,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:31,991 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:41,086 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 819 for changeToCandidate datanode_3 | 2019-10-15 03:17:31,992 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,277 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 703 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:31,992 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:24,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:32,109 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1188 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:41,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection728: begin an election at term 820 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:24,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:41,114 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection728: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t820, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:24,382 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 699 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:32,109 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection728: begin an election at term 821 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:24,382 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:32,109 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,148 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection728: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t821, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t821] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t821, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:24,382 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:32,109 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,148 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 821 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:24,382 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:32,248 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1189 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:41,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:24,425 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_3 | 2019-10-15 03:17:32,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:24,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:32,248 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,207 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:24,426 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 703 for changeToCandidate datanode_3 | 2019-10-15 03:17:32,248 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:32,303 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1190 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,426 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:32,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection630: begin an election at term 704 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:24,444 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection630 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:32,303 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:32,303 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:32,489 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1191 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:32,489 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:32,489 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:32,489 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:32,671 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1192 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:32,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:32,671 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:32,671 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:32,774 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1193 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:32,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:32,774 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:32,774 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:41,213 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_3 | 2019-10-15 03:17:32,954 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1194 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:41,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:32,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:41,213 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 821 for changeToCandidate datanode_3 | 2019-10-15 03:17:32,954 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:41,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:32,954 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:41,227 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection729: begin an election at term 822 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:33,124 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1195 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:33,124 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,242 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection729: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t822, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t822] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t822, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:33,124 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,242 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 822 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:41,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:33,124 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:41,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:33,212 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1196 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,462 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection630: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t704] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t704, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:41,304 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:33,212 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,462 INFO impl.LeaderElection: Exception 0: {} datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:33,212 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:33,212 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:33,444 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1197 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:33,444 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:33,444 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:33,444 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:33,629 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1198 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:33,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:33,629 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:33,629 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:33,851 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1199 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:33,851 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:41,364 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:33,851 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:33,851 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:33,914 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1200 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:33,914 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:33,914 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:33,914 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:24,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 704 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:34,071 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1201 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:24,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:34,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:24,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:34,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 700 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:34,071 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:34,236 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1202 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:24,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:34,236 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:24,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:34,236 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:24,605 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:17:34,236 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:24,605 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:34,432 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1203 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,605 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 704 for changeToCandidate datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:34,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,605 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:34,432 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,618 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection631: begin an election at term 705 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:34,433 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,627 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection631 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:41,442 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_3 | 2019-10-15 03:17:34,534 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1204 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:41,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:34,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 822 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:34,534 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:34,534 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,457 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection730: begin an election at term 823 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:41,458 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:34,560 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1205 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:34,560 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:34,560 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:34,560 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:34,672 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1206 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:34,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:34,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:34,672 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:34,755 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1207 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:34,755 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:34,755 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:34,755 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:34,804 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1208 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:34,804 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:34,804 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,633 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection631: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t705] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t705, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:34,804 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:24,633 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:35,031 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1209 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:35,031 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,485 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection730: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t823, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t823] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t823, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:35,031 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 823 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:35,031 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:35,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1210 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:41,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:35,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,605 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:35,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:35,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:35,393 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1211 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:35,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:35,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:35,393 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:35,419 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1212 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:35,419 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:35,419 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:35,419 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:35,604 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1213 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:35,604 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:35,604 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:35,604 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:35,754 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1214 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:35,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:35,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,651 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | 2019-10-15 03:16:24,633 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 705 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:35,754 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:35,810 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1215 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:41,651 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 823 for changeToCandidate datanode_2 | 2019-10-15 03:16:24,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:35,810 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,651 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:24,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 701 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:35,811 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,666 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection731: begin an election at term 824 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:24,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection731: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t824, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t824] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t824, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:35,811 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:41,695 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 824 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:24,713 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:35,848 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1216 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,790 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_3 | 2019-10-15 03:17:35,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:41,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:24,791 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:35,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:24,791 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 705 for changeToCandidate datanode_1 | 2019-10-15 03:16:41,820 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:35,848 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,791 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:36,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1217 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection632: begin an election at term 706 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:36,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,808 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection632 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:36,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:36,006 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:36,158 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1218 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:36,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:36,158 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:36,158 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:36,186 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1219 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:36,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:36,186 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:36,186 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:36,373 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1220 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:36,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:36,374 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:36,374 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,878 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:36,554 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1221 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:41,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:36,554 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:36,554 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:41,878 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 824 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:36,554 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:41,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:41,880 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:36,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1222 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:36,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:36,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:36,677 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:24,820 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection632: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t706] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t706, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:36,886 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1223 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:24,820 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:36,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:36,886 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:36,886 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:37,040 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1224 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:37,040 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:37,040 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:37,040 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:37,190 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1225 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:37,190 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:37,191 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:37,191 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:37,322 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1226 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:41,892 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection732: begin an election at term 825 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:41,921 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection732: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t825, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t825] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t825, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:41,921 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 825 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:37,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:41,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:37,322 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,921 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:37,322 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:41,995 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | 2019-10-15 03:17:37,479 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1227 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:41,995 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:37,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:41,995 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 825 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:37,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:41,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:37,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,008 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:37,685 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1228 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:37,685 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:37,685 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:24,820 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 706 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:37,685 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:24,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:37,788 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1229 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:24,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:37,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:24,871 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_3 | 2019-10-15 03:17:37,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:24,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:37,788 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,871 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 706 for changeToCandidate datanode_3 | 2019-10-15 03:17:37,941 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1230 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,871 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:37,941 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,882 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection633: begin an election at term 707 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:37,941 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,883 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 702 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:37,941 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:24,883 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:38,136 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1231 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,883 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:24,883 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:38,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:24,886 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection633 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:38,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:38,136 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:38,270 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1232 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:38,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,010 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection733: begin an election at term 826 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:38,270 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,026 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection733: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t826, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:38,270 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection733: begin an election at term 827 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:38,287 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1233 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,055 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection733: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t827, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t827] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t827, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:38,287 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:38,287 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 827 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:38,287 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:42,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:42,144 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:38,463 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1234 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:38,463 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:38,463 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:38,463 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:38,632 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1235 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:38,632 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:38,632 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:38,632 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:38,800 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1236 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:38,800 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:38,800 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection633: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t707] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t707, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:24,899 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:38,800 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:38,848 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1237 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:38,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:38,848 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:38,848 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,221 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:38,912 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1238 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:38,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:38,912 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:38,912 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:39,081 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1239 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:39,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:39,082 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:39,082 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:39,297 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1240 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:39,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:39,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:39,297 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:39,544 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1241 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:39,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:39,544 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:39,544 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:39,610 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1242 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:24,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 707 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:42,232 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_3 | 2019-10-15 03:17:39,610 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:24,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:42,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:39,610 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:24,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:42,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 827 for changeToCandidate datanode_3 | 2019-10-15 03:17:39,610 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:24,988 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_3 | 2019-10-15 03:17:39,836 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1243 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,308 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:24,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:39,836 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:39,836 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:24,988 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 707 for changeToCandidate datanode_3 | 2019-10-15 03:17:39,836 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:24,988 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:40,032 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1244 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,009 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection634: begin an election at term 708 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:40,032 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:25,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 703 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:40,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:25,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:40,033 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:25,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:40,115 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1245 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:25,010 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:40,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:25,013 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection634 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:40,115 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:40,115 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:40,315 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1246 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:40,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:40,315 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:40,315 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:40,455 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1247 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:40,455 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:40,455 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,313 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection734: begin an election at term 828 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:40,455 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:40,581 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1248 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,330 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection734: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t828, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t828] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t828, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:40,581 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,330 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 828 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:40,581 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:40,581 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:40,693 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1249 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,469 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:40,693 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:40,693 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:40,694 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:40,785 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1250 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:40,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:40,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,024 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection634: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t708] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t708, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:40,785 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:25,024 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:41,018 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1251 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:41,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:41,018 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:41,018 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:41,084 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1252 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:41,084 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:41,084 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:41,084 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:41,194 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1253 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:41,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:41,194 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,480 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:41,194 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:41,298 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1254 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:41,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:41,298 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:41,298 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:41,325 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1255 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:41,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:41,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:41,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:41,465 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1256 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:41,465 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:41,465 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,025 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 708 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:41,465 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:41,583 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1257 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:41,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,225 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:41,583 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:41,583 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,493 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_2 | 2019-10-15 03:16:25,225 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 708 for changeToCandidate datanode_3 | 2019-10-15 03:17:41,766 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1258 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,493 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:41,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,232 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection635: begin an election at term 709 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:41,766 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,493 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 828 for changeToCandidate datanode_2 | 2019-10-15 03:16:25,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 704 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,493 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,507 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection735: begin an election at term 829 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,234 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:42,515 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection735: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t829, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t829] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t829, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,234 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:41,766 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,515 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 829 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:25,237 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection635 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:42,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:41,891 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1259 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:42,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:41,891 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:42,686 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_3 | 2019-10-15 03:17:41,892 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:42,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:41,892 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:42,686 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 829 for changeToCandidate datanode_3 | 2019-10-15 03:17:42,071 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1260 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:42,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:42,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:42,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection736: begin an election at term 830 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:42,071 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:42,699 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:42,071 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:42,141 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1261 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:42,141 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:42,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:42,142 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:42,231 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1262 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:42,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:42,231 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:42,231 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:42,260 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1263 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:42,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:42,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:42,260 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:42,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1264 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:42,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,248 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection635: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t709] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t709, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:42,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:42,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:25,248 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:42,563 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1265 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,704 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection736: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t830, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t830] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t830, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:42,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,704 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 830 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:42,563 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:42,563 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,704 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:42,723 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1266 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,718 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:42,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,718 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:42,723 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,718 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 830 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:42,723 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,718 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:42,802 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1267 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,724 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection737: begin an election at term 831 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:42,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,725 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:42,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:42,803 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:42,855 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1268 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:42,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:42,855 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:42,855 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:42,958 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1269 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:42,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:42,958 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:42,958 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:43,116 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1270 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:43,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,249 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 709 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:43,116 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:43,116 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:25,249 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:43,362 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1271 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:25,375 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_3 | 2019-10-15 03:17:43,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,755 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection737: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t831, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t831] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t831, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:43,362 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,755 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 831 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:25,375 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 709 for changeToCandidate datanode_3 | 2019-10-15 03:17:43,362 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,755 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:25,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:43,503 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1272 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,755 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:25,398 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection636: begin an election at term 710 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:43,503 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,809 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:25,400 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 705 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:43,503 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,400 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:43,503 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,400 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:43,653 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1273 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,403 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection636 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:43,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:43,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:43,653 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:43,680 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1274 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:43,680 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:43,681 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:43,681 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:43,770 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1275 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:43,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:43,770 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:43,770 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:43,961 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1276 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:43,961 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:43,961 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:43,961 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:42,864 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_3 | 2019-10-15 03:17:44,171 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1277 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:42,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:44,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,864 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 831 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:44,171 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,864 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:44,171 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,877 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection738: begin an election at term 832 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:44,376 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1278 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:42,905 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection738: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t832, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t832] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t832, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:44,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:42,905 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 832 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:44,376 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:42,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:44,376 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:42,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:44,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1279 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:42,944 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:44,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:44,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,426 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection636: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t710] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t710, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:44,404 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,426 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:44,506 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1280 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:44,506 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:44,506 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:44,506 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:44,547 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1281 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:44,547 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:44,547 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:44,547 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:44,754 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1282 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:44,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:44,754 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:44,754 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:44,760 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1283 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:44,760 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:44,760 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,056 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:44,760 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:44,920 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1284 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:44,920 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:44,920 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:44,920 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:45,062 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1285 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:45,062 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:25,426 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 710 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:45,063 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,426 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:45,063 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,427 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:45,165 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1286 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,444 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:45,165 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,444 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:45,166 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,444 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 710 for changeToCandidate datanode_3 | 2019-10-15 03:17:45,166 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,444 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:45,205 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1287 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,458 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection637: begin an election at term 711 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:45,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,463 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection637 got exception when requesting votes: {} datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:45,205 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:43,106 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_3 | 2019-10-15 03:17:45,205 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:43,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:45,250 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1288 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,106 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 832 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:45,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:43,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:45,250 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:43,121 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection739: begin an election at term 833 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:45,250 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:43,150 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection739: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t833, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t833] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t833, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:45,459 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1289 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:45,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,150 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 833 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:45,459 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:43,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:45,459 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:43,240 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:45,659 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1290 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:45,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:45,659 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:45,659 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:45,863 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1291 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:45,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:45,863 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:45,863 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:46,010 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1292 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:46,010 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:46,010 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:46,010 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:25,483 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection637: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t711] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t711, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:46,088 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1293 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:25,483 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:46,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:46,088 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:46,088 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:46,273 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1294 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:46,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,279 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:46,273 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:46,273 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,279 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 833 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:43,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:46,338 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1295 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:43,310 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection740: begin an election at term 834 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:46,338 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:43,311 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:46,338 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:46,338 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:46,492 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1296 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:46,492 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:46,492 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:46,492 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:46,522 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1297 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:46,522 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:46,523 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:46,523 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:25,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 711 for DISCOVERED_A_NEW_TERM datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:46,630 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1298 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:46,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:46,631 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,507 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 706 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:25,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,326 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection740: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t834, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t834] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t834, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:46,631 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:43,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 834 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:46,748 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1299 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,508 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:46,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,542 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 707 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:25,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:46,748 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,445 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:25,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:46,748 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:25,543 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:46,893 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1300 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:46,893 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:46,893 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:25,647 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 711 for changeToCandidate datanode_3 | 2019-10-15 03:17:46,893 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:25,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:46,920 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1301 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:25,673 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection638: begin an election at term 712 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:46,920 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:25,677 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection638 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:46,921 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:46,921 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:47,006 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1302 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:47,006 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:47,007 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:47,007 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:47,217 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1303 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:47,217 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:47,217 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,508 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:47,217 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:47,396 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1304 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:47,396 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:47,396 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:47,396 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:47,514 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1305 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:47,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:47,514 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:47,514 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:47,601 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1306 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:47,601 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:47,601 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:47,601 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:47,830 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1307 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,703 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection638: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t712] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t712, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:47,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,703 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:47,830 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:43,523 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_3 | 2019-10-15 03:17:47,830 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:43,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:48,001 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1308 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:48,001 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,523 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 834 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:48,001 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,523 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:48,001 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,536 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection741: begin an election at term 835 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:48,210 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1309 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,567 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection741: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t835, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t835] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t835, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:48,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,567 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 835 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:48,210 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:48,210 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:48,325 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1310 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,571 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:48,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:48,325 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:48,326 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:48,415 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1311 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:48,415 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:48,415 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:48,415 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:48,454 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1312 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:25,704 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 712 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:48,454 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:25,704 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:48,454 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:25,705 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:48,454 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:25,752 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_3 | 2019-10-15 03:17:48,672 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1313 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:25,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:48,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:25,752 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 712 for changeToCandidate datanode_3 | 2019-10-15 03:17:48,672 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:25,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:48,672 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:25,754 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 708 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:48,789 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1314 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,629 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_2 | 2019-10-15 03:16:25,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:48,789 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:48,789 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 835 for changeToCandidate datanode_2 | 2019-10-15 03:16:25,754 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:48,789 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:25,778 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection639: begin an election at term 713 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:43,644 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection742: begin an election at term 836 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:48,858 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1315 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,782 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection639 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:48,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:48,858 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection742: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t836, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:48,858 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,670 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection742: begin an election at term 837 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:49,034 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1316 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection742: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t837, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t837] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t837, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:49,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 837 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:49,034 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:43,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:49,034 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:43,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:49,196 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1317 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:43,707 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_3 | 2019-10-15 03:17:49,196 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:49,196 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:17:49,196 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,707 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 837 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:49,247 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1318 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:43,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:43,721 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection743: begin an election at term 838 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:43,736 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection743: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t838, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t838] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t838, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:49,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:43,736 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 838 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:49,247 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:43,736 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:49,247 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:43,736 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:49,339 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1319 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:43,778 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:49,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:49,339 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:49,339 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:49,354 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1320 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:49,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:49,354 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:49,354 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:25,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection639: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t713] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t713, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:49,462 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1321 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:25,803 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:49,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:49,462 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:49,462 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:49,615 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1322 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:49,615 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:49,615 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:49,615 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,809 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:49,694 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1323 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:49,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:49,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:49,694 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:49,726 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1324 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:49,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:49,726 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:49,726 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:49,950 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1325 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:49,950 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:49,950 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:49,950 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:50,175 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1326 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:50,175 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:25,804 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 713 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:50,176 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:25,804 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:50,176 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,898 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_2 | 2019-10-15 03:16:25,804 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:50,403 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1327 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,856 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_3 | 2019-10-15 03:17:50,403 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:43,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,856 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:50,403 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,856 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 713 for changeToCandidate datanode_1 | 2019-10-15 03:16:43,898 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 838 for changeToCandidate datanode_2 | 2019-10-15 03:16:25,856 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:50,403 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:43,898 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:50,477 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1328 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,870 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection640: begin an election at term 714 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:43,924 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection744: begin an election at term 839 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:50,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,874 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection640 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:43,927 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:50,477 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:50,477 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:50,626 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1329 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:50,626 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:50,626 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:50,626 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:50,651 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1330 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:50,651 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:50,651 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:50,652 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:50,779 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1331 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:50,779 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:50,780 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:50,780 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:43,935 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection744: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t839, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:50,957 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1332 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:43,939 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection744: begin an election at term 840 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:50,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:43,966 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection744: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t840, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t840] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t840, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:50,957 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:43,966 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 840 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:50,957 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:43,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:51,011 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1333 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:43,966 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:51,011 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:44,109 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_3 | 2019-10-15 03:17:51,012 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:51,012 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:44,109 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 840 for changeToCandidate datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:51,065 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1334 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:51,065 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection745: begin an election at term 841 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:51,065 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,138 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:51,065 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:25,883 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection640: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t714] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t714, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:51,260 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1335 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:51,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,883 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:51,260 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:51,260 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:51,295 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1336 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:51,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:51,295 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:51,295 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:51,404 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1337 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:51,404 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:51,405 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:51,405 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:51,476 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1338 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:51,476 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:51,476 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:51,476 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:44,150 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection745: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t841, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t841] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t841, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:51,587 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1339 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:51,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:44,150 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 841 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:51,587 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:44,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:51,587 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:44,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:51,785 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1340 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,187 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:51,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:51,785 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:25,883 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 714 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:51,785 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:25,883 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:51,955 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1341 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:25,883 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:51,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,902 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:51,955 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:51,955 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,902 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 714 for changeToCandidate datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:51,999 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1342 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:25,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:51,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:25,916 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection641: begin an election at term 715 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:51,999 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | 2019-10-15 03:16:25,918 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 709 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:51,999 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:25,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:52,081 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1343 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:52,081 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:52,081 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,918 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:52,081 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,921 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection641 got exception when requesting votes: {} datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:52,096 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1344 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:52,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:44,225 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:52,096 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:52,096 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:52,226 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1345 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:52,226 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:52,226 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:52,227 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:52,379 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1346 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:52,379 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:52,380 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:52,380 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:52,501 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1347 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:44,240 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:52,501 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:52,501 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:52,501 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:52,714 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1348 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:52,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:52,715 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:52,715 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,933 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection641: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t715] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t715, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:52,890 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1349 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,933 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:52,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:52,890 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:52,890 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:53,127 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1350 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:53,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:53,127 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:53,127 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:53,250 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1351 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:44,262 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:53,251 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:53,251 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:53,251 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:53,328 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1352 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:53,328 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:53,328 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:53,328 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:53,422 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1353 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:53,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:17:53,422 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:53,422 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:53,593 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1354 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:53,593 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:53,593 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:44,323 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_3 | 2019-10-15 03:17:53,593 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:44,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:53,609 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1355 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,933 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 715 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:44,324 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 841 for changeToCandidate datanode_3 | 2019-10-15 03:17:53,609 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:25,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:44,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:25,933 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:53,609 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,337 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection746: begin an election at term 842 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,963 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_1 | 2019-10-15 03:16:44,358 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection746: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t842, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t842] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t842, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:53,609 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:25,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,358 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 842 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:53,756 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1356 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:25,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 715 for changeToCandidate datanode_1 | 2019-10-15 03:16:44,358 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:25,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:53,756 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:25,977 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection642: begin an election at term 716 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:53,756 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,388 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_2 | 2019-10-15 03:16:25,980 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection642 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:53,756 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:44,388 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:53,927 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1357 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,388 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 842 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:53,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,388 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:53,927 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,401 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection747: begin an election at term 843 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:44,419 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection747: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t843, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t843] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t843, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:44,419 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 843 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:53,927 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:44,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:54,057 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1358 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,057 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,482 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:54,057 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:54,057 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:54,182 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: addNew group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-BD8D864B3B3A:java.util.concurrent.CompletableFuture@36816429[Not completed] datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:54,185 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: new RaftServerImpl for group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:54,186 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:54,187 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:54,187 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:54,187 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a does not exist. Creating ... datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | 2019-10-15 03:16:25,991 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection642: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t716] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t716, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:54,201 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/in_use.lock acquired by nodename 7@074e53f4bc48 datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:25,991 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:54,214 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a has been successfully formatted. datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,214 [pool-8-thread-1] INFO - group-BD8D864B3B3A: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_1 | 2019-10-15 03:16:44,495 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:54,216 INFO segmented.SegmentedRaftLogWorker: new cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:54,217 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_1 | 2019-10-15 03:16:44,511 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_1 | 2019-10-15 03:16:44,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_1 | 2019-10-15 03:16:44,511 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 843 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_1 | 2019-10-15 03:16:44,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_1 | 2019-10-15 03:16:44,526 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection748: begin an election at term 844 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,991 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 716 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:54,222 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:44,535 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection748: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t844, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t844] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t844, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:25,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:54,223 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_1 | 2019-10-15 03:16:44,536 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 844 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:54,223 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:25,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:44,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:54,223 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-BD8D864B3B3A,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | 2019-10-15 03:16:26,014 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 710 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:54,253 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | 2019-10-15 03:16:26,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,573 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:54,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,014 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,253 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,014 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:54,253 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderElection datanode_2 | 2019-10-15 03:16:26,030 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 711 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:54,267 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-LeaderElection5: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:54,297 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1359 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,030 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_3 | 2019-10-15 03:17:54,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,030 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:54,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:26,181 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_3 | 2019-10-15 03:17:54,297 INFO impl.LeaderElection: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-LeaderElection5: Election PASSED; received 1 response(s) [cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:OK-t1] and 0 exception(s); cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A:t1, leader=null, voted=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:26,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:54,297 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | 2019-10-15 03:16:26,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 716 for changeToCandidate datanode_3 | 2019-10-15 03:17:54,297 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown LeaderElection datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:54,298 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:54,298 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: change Leader from null to cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at term 1 for becomeLeader, leader elected after 83ms datanode_2 | 2019-10-15 03:16:26,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:54,298 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) datanode_2 | 2019-10-15 03:16:26,211 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection643: begin an election at term 717 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | 2019-10-15 03:16:26,213 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 712 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | 2019-10-15 03:16:26,213 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:54,299 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | 2019-10-15 03:16:26,213 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:54,299 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:26,216 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection643 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:54,299 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) datanode_1 | 2019-10-15 03:16:44,590 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,299 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:54,299 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:54,302 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:54,302 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:54,302 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_3 | 2019-10-15 03:17:54,302 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:54,302 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:54,303 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,305 INFO server.RaftServerConfigKeys: raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:54,305 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:54,305 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.element-limit = 1 (custom) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:54,305 INFO grpc.GrpcConfigKeys$Server: raft.grpc.server.leader.outstanding.appends.max = 128 (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:54,306 INFO server.RaftServerConfigKeys: raft.server.rpc.request.timeout = 3000ms (default) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:54,306 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:54,308 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start LeaderState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:54,308 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-SegmentedRaftLogWorker: Starting segment from index:0 datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:17:54,310 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 datanode_1 | 2019-10-15 03:16:44,718 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:54,335 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1360 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:54,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:54,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:54,335 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:54,347 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-BD8D864B3B3A-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/current/log_inprogress_0 datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:54,460 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1361 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:26,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection643: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t717] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t717, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:54,460 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:54,460 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:26,239 INFO impl.LeaderElection: Exception 0: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:54,460 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:54,494 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1362 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:54,494 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:54,494 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:54,494 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:54,713 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1363 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:54,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:44,720 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_3 | 2019-10-15 03:17:54,714 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:44,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:54,714 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:44,720 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 844 for changeToCandidate datanode_3 | 2019-10-15 03:17:54,910 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1364 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:44,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:54,910 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:44,734 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection749: begin an election at term 845 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:44,763 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection749: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t845, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t845] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t845, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:44,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 845 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:54,910 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:44,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:54,910 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:44,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:55,121 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1365 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,798 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_3 | 2019-10-15 03:17:55,121 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:55,121 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:44,798 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 845 for changeToCandidate datanode_3 | 2019-10-15 03:17:55,121 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:44,798 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:55,335 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1366 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:44,812 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection750: begin an election at term 846 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:55,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:44,829 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection750: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t846, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t846] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t846, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:55,335 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:44,829 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 846 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:55,335 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:44,829 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:55,381 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1367 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:44,829 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:26,239 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 717 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:55,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:44,906 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:26,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:55,381 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:26,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:55,381 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | 2019-10-15 03:16:26,313 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_3 | 2019-10-15 03:17:55,533 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1368 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:55,533 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | 2019-10-15 03:16:26,313 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 717 for changeToCandidate datanode_3 | 2019-10-15 03:17:55,533 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:26,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:55,533 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:26,327 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection644: begin an election at term 718 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:55,653 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1369 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,332 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection644 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_3 | 2019-10-15 03:17:55,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:55,653 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:55,653 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_3 | 2019-10-15 03:17:55,728 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1370 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:55,728 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:55,728 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:55,728 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_3 | 2019-10-15 03:17:55,844 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1371 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:55,844 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:55,845 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:55,845 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:45,017 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:56,059 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1372 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:45,018 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:17:56,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:45,018 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 846 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:56,059 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:17:56,059 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:45,018 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:56,108 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1373 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:45,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection751: begin an election at term 847 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:56,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:45,060 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection751: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t847, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t847] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t847, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:56,108 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:56,108 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:45,060 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 847 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:56,280 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1374 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:45,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:56,280 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:45,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:56,280 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:45,118 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_2 | 2019-10-15 03:16:26,342 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection644: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t718] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t718, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:56,280 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | 2019-10-15 03:16:26,343 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:56,425 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1375 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:56,425 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:56,425 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:56,425 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:56,622 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1376 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:56,622 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_3 | 2019-10-15 03:17:56,622 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:17:56,622 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:17:56,945 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1377 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:56,945 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:45,239 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:45,239 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:45,239 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 847 for changeToCandidate datanode_3 | 2019-10-15 03:17:56,945 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:45,239 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:56,945 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:45,258 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection752: begin an election at term 848 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:57,068 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1378 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:57,068 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,343 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 718 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:45,259 WARN server.GrpcServerProtocolService: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: Failed requestVote 749b19c7-0772-44d2-8efd-0664e6aa0748->5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0 datanode_3 | 2019-10-15 03:17:57,068 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | org.apache.ratis.protocol.GroupMismatchException: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:57,068 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:122) datanode_3 | 2019-10-15 03:17:57,091 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1379 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,357 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 713 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:57,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,357 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:269) datanode_3 | 2019-10-15 03:17:57,092 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:278) datanode_2 | 2019-10-15 03:16:26,357 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:57,092 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:273) datanode_2 | 2019-10-15 03:16:26,357 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:17:57,148 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1380 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:443) datanode_2 | 2019-10-15 03:16:26,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_1 | at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:142) datanode_3 | 2019-10-15 03:17:57,148 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:57,148 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) datanode_2 | 2019-10-15 03:16:26,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 718 for changeToCandidate datanode_3 | 2019-10-15 03:17:57,148 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) datanode_2 | 2019-10-15 03:16:26,388 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:57,340 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1381 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,411 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection645: begin an election at term 719 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) datanode_3 | 2019-10-15 03:17:57,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,415 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection645 got exception when requesting votes: {} datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:710) datanode_3 | 2019-10-15 03:17:57,340 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) datanode_3 | 2019-10-15 03:17:57,341 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:57,385 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1382 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:57,385 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:17:57,385 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:17:57,385 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:17:57,622 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1383 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:45,331 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection752: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t848, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:17:57,622 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:49,817 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection752: begin an election at term 849 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:49,834 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection752: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t849, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t849] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t849, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:57,622 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:57,622 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:49,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 849 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:57,802 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1384 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:49,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:57,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:49,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:57,802 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:49,875 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_3 | 2019-10-15 03:17:57,802 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:49,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:57,946 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1385 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:49,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 849 for changeToCandidate datanode_3 | 2019-10-15 03:17:57,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:49,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:57,946 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:49,889 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection753: begin an election at term 850 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:17:57,947 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:49,905 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection753: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t850, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t850] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t850, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:58,050 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1386 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:49,905 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 850 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:17:58,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:49,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:17:58,050 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:49,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:58,051 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,088 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | 2019-10-15 03:16:26,441 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection645: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t719] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t719, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,239 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1387 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,441 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:58,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,088 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 850 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:58,239 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:50,088 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:58,239 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,102 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection754: begin an election at term 851 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:50,118 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection754: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t851, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t851] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t851, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,343 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1388 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:50,118 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 851 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:58,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:50,118 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:58,343 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:50,118 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:58,343 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:50,144 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:50,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:58,488 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1389 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:50,144 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 851 for changeToCandidate datanode_1 | 2019-10-15 03:16:50,144 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:17:58,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,158 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection755: begin an election at term 852 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:50,174 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection755: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t852, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t852] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t852, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,488 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:50,174 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 852 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:58,488 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:50,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:58,635 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1390 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:50,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:58,635 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:50,220 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_3 | 2019-10-15 03:17:58,636 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:50,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:58,636 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:50,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 852 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:17:58,694 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1391 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:17:58,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:50,233 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: begin an election at term 853 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,694 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:50,235 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t853, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,694 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:17:58,806 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1392 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,441 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 719 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:58,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,248 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: begin an election at term 854 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,441 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:50,253 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t854, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,806 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:50,275 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: begin an election at term 855 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,806 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 714 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,289 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t855, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,865 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1393 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,303 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: begin an election at term 856 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:50,309 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection756: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t856, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t856] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t856, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:58,865 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,442 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,309 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 856 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:58,866 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:26,469 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 715 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:59,033 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1394 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:26,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:59,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,445 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_3 | 2019-10-15 03:17:59,033 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:50,445 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,469 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,445 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 856 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,587 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 716 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:17:59,033 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,445 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:26,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:17:59,202 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1395 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection757: begin an election at term 857 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:17:59,202 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,475 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection757: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t857, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t857] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t857, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,202 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,588 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,475 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 857 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:59,202 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,621 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_1 | 2019-10-15 03:16:50,475 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:59,369 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1396 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,621 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,475 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:59,369 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,621 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 719 for changeToCandidate datanode_1 | 2019-10-15 03:16:50,554 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_3 | 2019-10-15 03:17:59,369 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,621 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:17:59,370 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,635 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection646: begin an election at term 720 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:50,554 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:59,387 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1397 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,639 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection646 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:17:59,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,554 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 857 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:17:59,387 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:50,554 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:17:59,387 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,567 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection758: begin an election at term 858 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:17:59,546 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1398 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:50,583 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection758: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t858, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t858] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t858, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,546 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:50,583 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 858 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:59,546 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:50,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:59,546 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:50,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:59,611 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1399 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:50,644 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_3 | 2019-10-15 03:17:59,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:17:59,611 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:50,644 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 858 for changeToCandidate datanode_3 | 2019-10-15 03:17:59,611 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:50,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:50,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection759: begin an election at term 859 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,640 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1400 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:50,666 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection759: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t859, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t859] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t859, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:50,666 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 859 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:59,640 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:50,666 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:59,640 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:50,667 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:50,675 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_3 | 2019-10-15 03:17:59,837 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1401 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:50,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:59,838 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:50,675 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 859 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:50,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:50,682 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection760: begin an election at term 860 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,838 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,651 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection646: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t720] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t720, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:50,687 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection760: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t860, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,838 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection760: begin an election at term 861 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,652 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:17:59,876 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1402 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,713 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection760: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t861, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t861] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t861, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:50,713 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 861 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:17:59,876 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:50,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:17:59,876 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:50,713 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:17:59,876 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:50,788 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_3 | 2019-10-15 03:17:59,935 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1403 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:50,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:17:59,935 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:50,788 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 861 for changeToCandidate datanode_3 | 2019-10-15 03:17:59,935 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:50,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:50,801 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection761: begin an election at term 862 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:17:59,935 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:50,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection761: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t862, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t862] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t862, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:00,136 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1404 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:50,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 862 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:00,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:00,136 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:50,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:00,136 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,960 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:00,286 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1405 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,960 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:50,960 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 862 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:50,961 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:00,286 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:50,974 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection762: begin an election at term 863 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:18:00,287 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:50,990 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection762: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t863, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t863] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t863, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:00,287 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:50,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 863 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:00,436 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1406 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:50,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:00,436 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:50,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:00,436 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:50,993 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_3 | 2019-10-15 03:18:00,436 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,652 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 720 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:50,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:00,473 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1407 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,652 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:50,993 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 863 for changeToCandidate datanode_3 | 2019-10-15 03:18:00,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,652 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:50,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:00,473 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_1 | 2019-10-15 03:16:51,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection763: begin an election at term 864 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:00,474 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,021 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection763: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t864, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t864] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t864, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:00,512 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1408 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:51,021 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 864 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:26,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 720 for changeToCandidate datanode_1 | 2019-10-15 03:16:51,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:00,512 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,021 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:00,512 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,210 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_2 | 2019-10-15 03:16:26,688 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:18:00,512 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,714 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection647: begin an election at term 721 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:00,551 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1409 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:51,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 864 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,716 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 717 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:18:00,551 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:00,551 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,265 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection764: begin an election at term 865 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,716 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:18:00,551 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,282 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection764: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t865, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t865] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t865, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,716 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,283 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 865 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:00,762 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1410 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,718 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection647 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:51,283 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:00,762 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:51,283 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:00,763 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:51,346 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_3 | 2019-10-15 03:18:00,763 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:51,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:00,940 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1411 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:51,346 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 865 for changeToCandidate datanode_3 | 2019-10-15 03:18:00,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:51,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:00,940 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:51,360 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection765: begin an election at term 866 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:00,940 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:51,376 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection765: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t866, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t866] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t866, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,100 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1412 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:51,376 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 866 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:01,100 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:01,100 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:01,100 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,573 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:01,181 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1413 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:51,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:01,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,573 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 866 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:01,181 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:01,181 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,587 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection766: begin an election at term 867 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,274 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1414 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:51,603 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection766: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t867, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t867] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t867, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:51,604 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 867 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:51,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:01,274 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:51,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:01,275 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:51,747 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_3 | 2019-10-15 03:18:01,314 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1415 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:51,747 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:01,314 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,740 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection647: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t721] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t721, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:51,747 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 867 for changeToCandidate datanode_3 | 2019-10-15 03:18:01,314 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,740 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:51,747 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:01,314 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:51,770 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection767: begin an election at term 868 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,470 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1416 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:51,780 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection767: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t868, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t868] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t868, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,780 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 868 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:01,470 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:51,780 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:01,470 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:51,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:01,587 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1417 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:51,933 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_3 | 2019-10-15 03:18:01,588 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:01,588 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 868 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:18:01,588 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:51,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:18:01,604 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1418 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:51,957 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection768: begin an election at term 869 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:01,604 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:51,986 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection768: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t869, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t869] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t869, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:01,604 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:51,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 869 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:01,604 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:51,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:01,792 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1419 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:51,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:01,792 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:52,058 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_3 | 2019-10-15 03:18:01,792 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:52,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:01,792 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:52,059 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 869 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:01,897 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1420 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:01,897 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,076 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection769: begin an election at term 870 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,897 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:52,104 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection769: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t870, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t870] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t870, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:01,897 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:52,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 870 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:02,053 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1421 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:02,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:02,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,172 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_3 | 2019-10-15 03:18:02,053 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:52,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,740 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 721 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:02,095 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1422 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 870 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:02,095 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:52,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:02,095 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_1 | 2019-10-15 03:16:52,187 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection770: begin an election at term 871 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:02,095 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:02,220 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1423 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 721 for changeToCandidate datanode_3 | 2019-10-15 03:18:02,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:52,212 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection770: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t871, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t871] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t871, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:02,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:26,863 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection648: begin an election at term 722 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:52,212 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 871 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:02,220 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:26,867 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection648 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:52,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:02,349 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1424 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,212 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:02,349 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:52,275 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_3 | 2019-10-15 03:18:02,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,275 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:02,350 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:52,275 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 871 for changeToCandidate datanode_3 | 2019-10-15 03:18:02,442 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1425 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:52,275 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:02,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:52,288 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection771: begin an election at term 872 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:02,443 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:52,297 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection771: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t872, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t872] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t872, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:02,443 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:52,297 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 872 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:02,643 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1426 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:52,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:02,643 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:02,643 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,426 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:02,643 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:52,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:02,821 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1427 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,426 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 872 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:02,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,426 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:02,822 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,440 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection772: begin an election at term 873 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:02,822 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:52,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection772: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t873, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t873] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t873, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,053 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1428 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,459 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 873 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:03,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:03,053 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:18:03,053 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:52,625 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:03,142 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1429 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:03,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,625 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 873 for changeToCandidate datanode_3 | 2019-10-15 03:18:03,142 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:52,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:03,142 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:52,638 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection773: begin an election at term 874 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,225 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1430 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection773: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t874, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,879 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection648: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t722] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t722, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,225 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection773: begin an election at term 875 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,879 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:18:03,225 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,696 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection773: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t875, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t875] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t875, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:03,226 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:52,697 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 875 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:18:03,241 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1431 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:18:03,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:52,697 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:18:03,241 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:52,896 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:18:03,241 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:52,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:18:03,393 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1432 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,896 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 875 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:52,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:03,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:52,910 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection774: begin an election at term 876 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,393 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:52,937 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection774: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t876, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t876] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t876, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,393 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:52,937 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 876 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:03,619 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1433 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:52,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:03,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:52,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:03,619 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:03,619 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,116 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:03,660 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1434 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,116 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:03,660 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,116 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 876 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:03,660 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,116 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:03,660 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,142 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection775: begin an election at term 877 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:53,159 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection775: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t877, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t877] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t877, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,788 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1435 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:53,159 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 877 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:03,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:53,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:03,788 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:53,159 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:18:03,788 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,200 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_1 | 2019-10-15 03:16:53,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 722 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:53,200 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 877 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:53,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:03,832 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1436 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:26,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:53,214 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection776: begin an election at term 878 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:03,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,231 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection776: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t878, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t878] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t878, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:26,899 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 718 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:18:03,832 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,231 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 878 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:26,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:03,832 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:26,899 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:18:04,035 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1437 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:26,899 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_3 | 2019-10-15 03:18:04,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,409 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:16:26,966 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:18:04,035 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:26,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:04,035 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,409 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 878 for changeToCandidate datanode_2 | 2019-10-15 03:16:26,966 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 722 for changeToCandidate datanode_3 | 2019-10-15 03:18:04,150 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1438 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,409 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:26,966 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_3 | 2019-10-15 03:18:04,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection777: begin an election at term 879 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,001 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: begin an election at term 723 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:53,477 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection777: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t879, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t879] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t879, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,002 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t723, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:04,150 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:27,027 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: begin an election at term 724 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:53,477 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 879 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:04,150 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,477 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:53,477 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:04,169 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1439 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,633 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_3 | 2019-10-15 03:18:04,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 719 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:04,169 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_3 | 2019-10-15 03:18:04,169 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,633 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 879 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:04,333 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1440 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,633 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,031 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:18:04,333 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,646 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection778: begin an election at term 880 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:04,333 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:53,662 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection778: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t880, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t880] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t880, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:04,333 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:53,662 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 880 for DISCOVERED_A_NEW_TERM datanode_3 | 2019-10-15 03:18:04,479 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1441 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:53,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:04,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:53,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:04,479 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:53,762 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_3 | 2019-10-15 03:18:04,479 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:18:04,637 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1442 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,763 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 880 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:18:04,637 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:04,637 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,788 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection779: begin an election at term 881 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:04,637 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:53,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection779: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t881, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t881] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t881, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:04,767 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1443 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:53,804 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 881 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:04,767 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:53,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:04,767 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:53,804 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:04,767 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,000 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:04,982 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1444 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,000 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:04,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,000 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 881 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:04,983 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,016 [Thread-4126] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120197ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:18:04,983 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:05,110 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1445 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,029 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection780: begin an election at term 882 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:05,110 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:05,110 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,031 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: addNew group-2AAF8832D47C:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858] returns group-2AAF8832D47C:java.util.concurrent.CompletableFuture@2e031ef1[Not completed] datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:54,034 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: new RaftServerImpl for group-2AAF8832D47C:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858] with ContainerStateMachine:uninitialized datanode_3 | 2019-10-15 03:18:05,110 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:27,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,034 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_3 | 2019-10-15 03:18:05,191 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1446 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:27,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_3 | 2019-10-15 03:18:05,191 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,037 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t724, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_3 | 2019-10-15 03:18:05,191 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:27,037 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_3 | 2019-10-15 03:18:05,191 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:54,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null, confs= datanode_3 | 2019-10-15 03:18:05,206 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1447 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_3 | 2019-10-15 03:18:05,206 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:18:05,206 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:54,035 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_3 | 2019-10-15 03:18:05,206 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:54,036 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/b0c87df0-d20c-4205-986d-2aaf8832d47c does not exist. Creating ... datanode_3 | 2019-10-15 03:18:05,370 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1448 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:54,057 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/b0c87df0-d20c-4205-986d-2aaf8832d47c/in_use.lock acquired by nodename 7@2783c458fa57 datanode_3 | 2019-10-15 03:18:05,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,058 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection780: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t882, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t882] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t882, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:05,370 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:05,370 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:05,427 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1449 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,058 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 882 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:05,427 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,058 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:05,427 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:05,427 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,070 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/b0c87df0-d20c-4205-986d-2aaf8832d47c has been successfully formatted. datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:05,556 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1450 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,071 [pool-8-thread-1] INFO - group-2AAF8832D47C: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:05,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,071 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:05,556 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,071 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:54,071 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_3 | 2019-10-15 03:18:05,556 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,071 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:05,647 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1451 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,071 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:05,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:18:05,647 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO segmented.SegmentedRaftLogWorker: new 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/b0c87df0-d20c-4205-986d-2aaf8832d47c datanode_2 | 2019-10-15 03:16:27,046 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: begin an election at term 725 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:05,647 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_2 | 2019-10-15 03:16:27,051 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:18:05,662 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1452 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:05,662 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:18:05,662 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:18:05,662 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:18:05,677 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1453 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:18:05,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:18:05,677 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:18:05,677 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:05,751 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1454 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,072 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:05,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,073 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:05,751 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,073 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:05,751 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,073 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:05,961 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1455 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,073 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:54,073 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_3 | 2019-10-15 03:18:05,961 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:54,079 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null datanode_3 | 2019-10-15 03:18:05,961 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:54,079 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_3 | 2019-10-15 03:18:05,961 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:06,042 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1456 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:54,079 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-2AAF8832D47C,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:18:06,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,238 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_3 | 2019-10-15 03:18:06,042 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:54,239 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:06,042 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,239 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:18:06,106 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1457 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,052 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection649: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t725] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t725, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_3 | 2019-10-15 03:18:06,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,249 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_2 | 2019-10-15 03:16:27,052 INFO impl.LeaderElection: Exception 0: {} datanode_3 | 2019-10-15 03:18:06,106 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:06,106 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,249 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 882 for changeToCandidate datanode_3 | 2019-10-15 03:18:06,188 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1458 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:54,250 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:06,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,250 [Thread-4130] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120438ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:18:06,188 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,257 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-LeaderElection781: begin an election at term 1 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_3 | 2019-10-15 03:18:06,188 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_3 | 2019-10-15 03:18:06,350 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1459 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,258 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_3 | 2019-10-15 03:18:06,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,258 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: change Leader from null to 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c at term 1 for becomeLeader, leader elected after 186ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_3 | 2019-10-15 03:18:06,350 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,258 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_3 | 2019-10-15 03:18:06,350 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,258 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:06,568 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1460 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,259 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_3 | 2019-10-15 03:18:06,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,259 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_3 | 2019-10-15 03:18:06,568 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,259 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_3 | 2019-10-15 03:18:06,568 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_3 | 2019-10-15 03:18:06,705 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1461 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:18:06,705 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_3 | 2019-10-15 03:18:06,705 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,259 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_3 | 2019-10-15 03:18:06,705 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_3 | 2019-10-15 03:18:06,879 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1462 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,264 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_3 | 2019-10-15 03:18:06,879 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,265 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858], old=null at 0 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_3 | 2019-10-15 03:18:06,880 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_1 | 2019-10-15 03:16:54,269 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection782: begin an election at term 883 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_3 | 2019-10-15 03:18:06,880 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,301 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection782: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t883, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t883] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t883, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,301 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 883 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_3 | 2019-10-15 03:18:06,953 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1463 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:54,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_3 | 2019-10-15 03:18:06,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_3 | 2019-10-15 03:18:06,953 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:27,052 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 725 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:54,314 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-2AAF8832D47C-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/b0c87df0-d20c-4205-986d-2aaf8832d47c/current/log_inprogress_0 datanode_3 | 2019-10-15 03:18:06,953 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:27,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:54,320 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_3 | 2019-10-15 03:18:07,089 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1464 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:27,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:18:07,090 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,189 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_1 | 2019-10-15 03:16:54,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_3 | 2019-10-15 03:18:07,090 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: start FollowerState datanode_2 | 2019-10-15 03:16:27,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,320 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 883 for changeToCandidate datanode_3 | 2019-10-15 03:18:07,090 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:27,189 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 725 for changeToCandidate datanode_1 | 2019-10-15 03:16:54,320 [Thread-4138] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120509ms datanode_3 | 2019-10-15 03:18:07,220 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: remove FOLLOWER cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13:t1464, leader=null, voted=null, raftlog=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f0,i0, conf=0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_2 | 2019-10-15 03:16:27,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:54,321 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_3 | 2019-10-15 03:18:07,220 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: shutdown datanode_1 | 2019-10-15 03:16:54,327 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection783: begin an election at term 884 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 720 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_3 | 2019-10-15 03:18:07,220 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_1 | 2019-10-15 03:16:54,343 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection783: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t884, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t884] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t884, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_3 | 2019-10-15 03:18:07,220 INFO impl.RoleInfo: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,344 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 884 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_3 | 2019-10-15 03:18:07,220 INFO impl.StateMachineUpdater: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-StateMachineUpdater: set stopIndex = -1 datanode_1 | 2019-10-15 03:16:54,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_3 | 2019-10-15 03:18:07,220 INFO impl.FollowerState: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:27,191 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:54,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_3 | 2019-10-15 03:18:07,221 INFO impl.RaftServerImpl: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13: closes. applyIndex: -1 datanode_2 | 2019-10-15 03:16:27,195 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection650: begin an election at term 726 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,430 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_3 | 2019-10-15 03:18:07,221 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | 2019-10-15 03:16:27,199 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection650 got exception when requesting votes: {} datanode_3 | 2019-10-15 03:18:07,222 INFO segmented.SegmentedRaftLogWorker: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3@group-D5F35BC43A13-SegmentedRaftLogWorker close() datanode_1 | 2019-10-15 03:16:54,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_3 | 2019-10-15 03:18:37,220 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: remove group-D5F35BC43A13:null datanode_1 | 2019-10-15 03:16:54,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 884 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_3 | 2019-10-15 03:19:01,699 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: remove group-D5F35BC43A13:null datanode_1 | 2019-10-15 03:16:54,431 [Thread-4142] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120619ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_3 | 2019-10-15 03:19:03,082 INFO impl.RaftServerProxy: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: remove group-D5F35BC43A13:null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:54,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:54,436 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection784: begin an election at term 885 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:54,446 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection784: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t885, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,452 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection784: begin an election at term 886 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:54,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection784: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t886, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t886] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t886, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,459 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 886 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:54,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:54,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:54,467 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:7ms, electionTimeout:7ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:54,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:54,467 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 886 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:54,467 [Thread-4148] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120656ms datanode_1 | 2019-10-15 03:16:54,467 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:54,481 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection785: begin an election at term 887 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,503 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection785: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t887, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t887] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t887, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:54,503 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 887 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:54,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:54,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,201 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection650: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t726] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t726, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,594 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_1 | 2019-10-15 03:16:54,594 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,594 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 887 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,201 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:54,594 [Thread-4152] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120783ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,595 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:54,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection786: begin an election at term 888 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:54,627 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection786: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t888, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t888] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t888, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:54,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 888 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:54,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:54,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:54,675 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:54,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:54,675 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 888 for changeToCandidate datanode_1 | 2019-10-15 03:16:54,675 [Thread-4156] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120864ms datanode_1 | 2019-10-15 03:16:54,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:54,701 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection787: begin an election at term 889 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,717 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection787: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t889, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t889] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t889, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:54,717 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 889 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:54,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:54,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:54,738 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_1 | 2019-10-15 03:16:54,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:54,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 889 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:54,738 [Thread-4160] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 120927ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:54,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:54,752 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection788: begin an election at term 890 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:54,778 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection788: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t890, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t890] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t890, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,201 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 726 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:54,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 890 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,201 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:54,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:54,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,291 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_1 | 2019-10-15 03:16:54,971 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | 2019-10-15 03:16:27,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:54,971 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,291 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 726 for changeToCandidate datanode_1 | 2019-10-15 03:16:54,971 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 890 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,291 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:54,971 [Thread-4164] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121160ms datanode_2 | 2019-10-15 03:16:27,316 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection651: begin an election at term 727 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:54,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,320 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection651 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:54,997 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection789: begin an election at term 891 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection789: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t891, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t891] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t891, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 891 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:55,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:55,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:55,093 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_1 | 2019-10-15 03:16:55,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:55,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 891 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:55,093 [Thread-4168] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121282ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:55,094 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,117 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection790: begin an election at term 892 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:55,134 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection790: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t892, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t892] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t892, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:55,134 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 892 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:55,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:55,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:55,187 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:55,187 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:55,187 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 892 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:55,187 [Thread-4172] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121376ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:55,187 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:55,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection791: begin an election at term 893 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:55,216 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection791: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t893, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t893] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t893, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:55,216 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 893 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:55,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,332 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection651: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t727] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t727, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,332 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:55,232 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:55,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 893 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:55,233 [Thread-4176] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121421ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:55,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:55,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection792: begin an election at term 894 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:55,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection792: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t894, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t894] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t894, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:55,261 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 894 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:55,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:55,346 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:55,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:55,346 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 894 for changeToCandidate datanode_1 | 2019-10-15 03:16:55,346 [Thread-4180] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121535ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:55,346 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:55,360 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection793: begin an election at term 895 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,378 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection793: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t895, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t895] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t895, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:55,378 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 895 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:55,378 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:55,378 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:55,393 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:55,393 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:55,393 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 895 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:55,393 [Thread-4184] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121582ms datanode_2 | 2019-10-15 03:16:27,332 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 727 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:55,394 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,332 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:55,407 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection794: begin an election at term 896 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 721 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:55,435 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection794: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t896, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t896] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t896, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:55,435 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 896 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:55,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:55,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,455 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_1 | 2019-10-15 03:16:55,512 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_2 | 2019-10-15 03:16:27,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,455 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 727 for changeToCandidate datanode_1 | 2019-10-15 03:16:55,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:55,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 896 for changeToCandidate datanode_1 | 2019-10-15 03:16:55,513 [Thread-4188] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121702ms datanode_1 | 2019-10-15 03:16:55,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,471 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection652: begin an election at term 728 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,542 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection795: begin an election at term 897 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,474 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection652 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,558 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection795: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t897, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t897] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t897, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:55,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 897 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:55,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:55,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:55,670 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:55,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:55,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 897 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:55,671 [Thread-4192] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 121860ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:55,671 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:55,694 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection796: begin an election at term 898 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:55,703 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection796: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t898, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:55,732 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection796: begin an election at term 899 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:55,749 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection796: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t899, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t899] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t899, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:55,749 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 899 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:55,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:55,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:55,909 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_1 | 2019-10-15 03:16:55,909 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:55,909 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 899 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:55,910 [Thread-4198] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122098ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:55,910 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:55,924 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection797: begin an election at term 900 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,952 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection797: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t900, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t900] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t900, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:55,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 900 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection652: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t728] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t728, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:55,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,477 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:55,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:56,123 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:56,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:56,123 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 900 for changeToCandidate datanode_1 | 2019-10-15 03:16:56,123 [Thread-4202] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122312ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:56,124 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:56,139 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection798: begin an election at term 901 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:56,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection798: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t901, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t901] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t901, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:56,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 901 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:56,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:56,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:56,314 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_1 | 2019-10-15 03:16:56,314 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:56,314 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 901 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:56,315 [Thread-4206] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122504ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:56,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:56,329 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection799: begin an election at term 902 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:56,344 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection799: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t902, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t902] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t902, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:56,344 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 902 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:56,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:56,344 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:56,503 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | 2019-10-15 03:16:27,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 728 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:56,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:56,503 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 902 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:56,503 [Thread-4210] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122692ms datanode_2 | 2019-10-15 03:16:27,614 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 722 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:56,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:56,517 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection800: begin an election at term 903 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:56,533 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection800: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t903, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t903] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t903, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,614 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:56,533 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 903 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,629 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_1 | 2019-10-15 03:16:56,533 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:56,533 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,629 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 728 for changeToCandidate datanode_1 | 2019-10-15 03:16:56,658 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | 2019-10-15 03:16:27,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:56,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,642 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection653: begin an election at term 729 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:56,658 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 903 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,645 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection653 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:56,658 [Thread-4214] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122847ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:56,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:56,672 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection801: begin an election at term 904 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:56,689 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection801: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t904] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t904, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:56,702 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection801: begin an election at term 905 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:56,720 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection801: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t905, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t905] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t905, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:56,720 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 905 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:56,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:56,720 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:56,738 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:56,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:56,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 905 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:56,738 [Thread-4220] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 122927ms datanode_1 | 2019-10-15 03:16:56,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:56,743 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection802: begin an election at term 906 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:56,759 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection802: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t906, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t906] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t906, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:56,759 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 906 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:56,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:56,759 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:56,917 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:56,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:56,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 906 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:56,918 [Thread-4224] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123107ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:56,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:56,932 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection803: begin an election at term 907 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:56,967 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection803: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t907, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t907] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t907, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:56,967 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 907 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,657 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection653: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t729] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t729, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,657 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:56,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:56,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:56,986 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:56,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:56,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 907 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:56,987 [Thread-4228] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123175ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:56,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:57,000 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection804: begin an election at term 908 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:57,015 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection804: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t908, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t908] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t908, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:57,015 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 908 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:57,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:57,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:57,055 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:57,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:57,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 908 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:57,055 [Thread-4232] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123244ms datanode_1 | 2019-10-15 03:16:57,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:57,069 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection805: begin an election at term 909 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:57,097 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection805: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t909, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t909] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t909, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:57,097 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 909 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:57,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,658 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 729 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:57,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,658 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,658 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:57,263 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_2 | 2019-10-15 03:16:27,742 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_1 | 2019-10-15 03:16:57,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:57,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 909 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,742 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:57,264 [Thread-4236] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123452ms datanode_1 | 2019-10-15 03:16:57,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,742 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 729 for changeToCandidate datanode_1 | 2019-10-15 03:16:57,278 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection806: begin an election at term 910 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,742 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:57,294 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection806: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t910, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t910] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t910, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,766 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection654: begin an election at term 730 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:57,295 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 910 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:57,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:27,770 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection654 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:57,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:57,331 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:57,331 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:57,331 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 910 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:57,332 [Thread-4240] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123520ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:57,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:57,357 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection807: begin an election at term 911 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:57,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection807: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t911, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t911] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t911, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:57,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 911 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:57,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:57,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:57,558 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:57,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:57,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 911 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:57,558 [Thread-4244] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123747ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:57,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:57,566 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection808: begin an election at term 912 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:57,576 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection808: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t912] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t912, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:57,584 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection808: begin an election at term 913 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:57,592 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection808: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t913, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t913] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t913, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:57,592 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 913 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:57,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:57,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:57,733 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | 2019-10-15 03:16:27,795 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection654: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t730] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t730, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:57,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:57,733 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 913 for changeToCandidate datanode_1 | 2019-10-15 03:16:57,733 [Thread-4250] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 123922ms datanode_2 | 2019-10-15 03:16:27,795 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:57,733 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:57,748 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection809: begin an election at term 914 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:57,763 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection809: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t914, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t914] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t914, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:57,763 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 914 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:57,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:57,763 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:57,886 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:57,887 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:57,887 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 914 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:57,887 [Thread-4254] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124076ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:57,887 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:57,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection810: begin an election at term 915 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:57,918 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection810: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t915, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t915] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t915, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:57,918 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 915 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:57,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:57,918 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,096 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_1 | 2019-10-15 03:16:58,096 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:58,096 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 915 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,096 [Thread-4258] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124285ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:58,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:58,111 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection811: begin an election at term 916 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:58,139 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection811: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t916, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t916] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t916, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:58,139 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 916 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:58,139 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:58,139 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:58,260 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 730 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:58,260 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:58,260 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 916 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:58,260 [Thread-4262] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124449ms datanode_1 | 2019-10-15 03:16:58,260 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 723 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:58,274 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection812: begin an election at term 917 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:58,292 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection812: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t917, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t917] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t917, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:27,795 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:58,292 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 917 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_1 | 2019-10-15 03:16:58,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:58,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:58,444 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | 2019-10-15 03:16:27,849 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 730 for changeToCandidate datanode_1 | 2019-10-15 03:16:58,444 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:58,444 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 917 for changeToCandidate datanode_2 | 2019-10-15 03:16:27,861 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection655: begin an election at term 731 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:58,444 [Thread-4266] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124633ms datanode_1 | 2019-10-15 03:16:58,445 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:58,470 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection813: begin an election at term 918 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,862 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection655: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t731, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,888 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection655: begin an election at term 732 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,892 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection655 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:58,477 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection813: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t918, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:58,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection813: begin an election at term 919 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:58,503 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection813: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t919, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t919] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t919, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:58,503 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 919 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:58,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:58,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:58,656 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:58,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:58,656 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 919 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:58,656 [Thread-4272] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124845ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:58,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:58,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection814: begin an election at term 920 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:58,700 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection814: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t920, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t920] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t920, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:58,700 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 920 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:58,700 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,701 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:16:58,722 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_1 | 2019-10-15 03:16:58,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:58,722 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 920 for changeToCandidate datanode_1 | 2019-10-15 03:16:58,722 [Thread-4276] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124911ms datanode_1 | 2019-10-15 03:16:58,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,751 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection815: begin an election at term 921 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:58,778 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection815: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t921, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t921] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t921, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:58,779 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 921 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:27,920 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection655: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t732] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t732, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:27,920 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:16:58,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:58,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:58,793 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:58,793 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:58,793 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 921 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:58,794 [Thread-4280] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 124982ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:58,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:58,807 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection816: begin an election at term 922 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:58,823 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection816: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t922, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t922] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t922, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:58,823 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 922 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:58,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:58,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:58,854 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_1 | 2019-10-15 03:16:58,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:58,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 922 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:58,855 [Thread-4284] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125043ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,855 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:58,869 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection817: begin an election at term 923 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:58,896 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection817: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t923, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t923] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t923, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:58,896 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 923 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:58,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:58,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:27,921 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 732 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:58,953 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_2 | 2019-10-15 03:16:27,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:58,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:27,921 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:28,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 724 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:58,953 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 923 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:58,953 [Thread-4288] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125142ms datanode_1 | 2019-10-15 03:16:58,953 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,006 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:28,033 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 725 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:16:58,967 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection818: begin an election at term 924 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,033 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:58,983 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection818: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t924, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t924] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t924, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,033 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:58,983 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 924 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,033 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:16:58,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,048 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | 2019-10-15 03:16:58,983 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:16:59,070 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | 2019-10-15 03:16:28,048 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 732 for changeToCandidate datanode_1 | 2019-10-15 03:16:59,070 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:59,070 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 924 for changeToCandidate datanode_1 | 2019-10-15 03:16:59,071 [Thread-4292] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125259ms datanode_2 | 2019-10-15 03:16:28,061 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection656: begin an election at term 733 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:59,071 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:59,094 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection819: begin an election at term 925 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,064 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection656 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:59,123 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection819: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t925, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t925] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t925, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:59,123 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 925 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:59,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:59,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:59,278 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:59,278 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:59,278 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 925 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:59,279 [Thread-4296] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125467ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:59,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:59,304 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection820: begin an election at term 926 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:59,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection820: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t926, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t926] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t926, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:59,320 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 926 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:59,320 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:59,321 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:59,470 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:59,470 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:16:59,470 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 926 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:59,470 [Thread-4300] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125659ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:59,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:59,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection821: begin an election at term 927 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:28,076 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection656: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t733] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t733, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:59,526 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection821: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t927, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t927] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t927, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,076 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:59,526 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 927 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:59,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:16:59,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:16:59,541 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:16:59,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:16:59,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 927 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:16:59,541 [Thread-4304] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125730ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:59,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:16:59,567 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection822: begin an election at term 928 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:16:59,583 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection822: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t928, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t928] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t928, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:59,583 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 928 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:16:59,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:16:59,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:16:59,764 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:16:59,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:16:59,765 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 928 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:16:59,765 [Thread-4308] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 125954ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:59,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:16:59,794 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection823: begin an election at term 929 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:16:59,810 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection823: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t929, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t929] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t929, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:16:59,810 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 929 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:16:59,810 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:16:59,810 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 733 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:16:59,946 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_2 | 2019-10-15 03:16:28,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:16:59,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:16:59,946 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 929 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,183 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:16:59,947 [Thread-4312] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126135ms datanode_2 | 2019-10-15 03:16:28,183 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,183 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 733 for changeToCandidate datanode_1 | 2019-10-15 03:16:59,947 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,183 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:16:59,960 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection824: begin an election at term 930 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,188 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection657: begin an election at term 734 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:16:59,975 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection824: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t930, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t930] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t930, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,192 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection657 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:16:59,975 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 930 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:16:59,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:16:59,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:00,105 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:00,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:00,106 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 930 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:00,106 [Thread-4316] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126295ms datanode_1 | 2019-10-15 03:17:00,106 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:00,120 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection825: begin an election at term 931 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,148 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection825: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t931, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t931] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t931, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:00,148 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 931 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:00,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:00,148 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:00,258 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:00,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:00,258 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 931 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:00,258 [Thread-4320] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126447ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:00,258 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:00,272 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection826: begin an election at term 932 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,289 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection826: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t932, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t932] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t932, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:00,290 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 932 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:00,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:00,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:00,338 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:00,338 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,204 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection657: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t734] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t734, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,338 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 932 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,204 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:00,339 [Thread-4324] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126527ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:00,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:00,351 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection827: begin an election at term 933 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,389 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection827: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t933, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t933] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t933, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:00,390 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 933 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:00,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:00,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:00,433 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:43ms, electionTimeout:43ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:00,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:00,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 933 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:00,433 [Thread-4328] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126622ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:00,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:00,448 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection828: begin an election at term 934 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:00,464 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection828: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t934, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t934] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t934, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:00,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 934 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:00,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:00,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:00,536 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:00,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:00,536 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 934 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:00,537 [Thread-4332] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126725ms datanode_2 | 2019-10-15 03:16:28,204 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 734 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:00,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,204 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:00,551 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection829: begin an election at term 935 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,205 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 726 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:00,567 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection829: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t935, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t935] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t935, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:00,567 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 935 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:00,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:00,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,205 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:00,698 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | 2019-10-15 03:16:28,308 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_1 | 2019-10-15 03:17:00,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:00,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 935 for changeToCandidate datanode_1 | 2019-10-15 03:17:00,698 [Thread-4336] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 126887ms datanode_2 | 2019-10-15 03:16:28,308 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 734 for changeToCandidate datanode_1 | 2019-10-15 03:17:00,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,308 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:00,724 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection830: begin an election at term 936 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,321 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection658: begin an election at term 735 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,751 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection830: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t936, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t936] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t936, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,751 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 936 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,325 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection658 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:00,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:00,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:00,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:00,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:00,842 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 936 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:00,843 [Thread-4340] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127031ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:00,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:00,868 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection831: begin an election at term 937 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:00,911 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection831: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t937, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t937] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t937, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:00,911 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 937 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:00,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:00,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:01,080 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:01,080 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:01,080 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 937 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:01,080 [Thread-4344] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127269ms datanode_1 | 2019-10-15 03:17:01,081 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:01,106 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection832: begin an election at term 938 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:01,122 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection832: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t938, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t938] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t938, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:01,122 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 938 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:01,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:01,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:01,249 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_1 | 2019-10-15 03:17:01,249 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:01,249 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 938 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,327 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection658: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t735] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t735, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:01,249 [Thread-4348] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127438ms datanode_1 | 2019-10-15 03:17:01,250 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,327 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:01,255 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection833: begin an election at term 939 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:01,263 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection833: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t939, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t939] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t939, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:01,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 939 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:01,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:01,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:01,390 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:01,390 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:01,390 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 939 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:01,391 [Thread-4353] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127579ms datanode_1 | 2019-10-15 03:17:01,391 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:01,398 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection834: begin an election at term 940 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:01,443 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection834: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t940, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:01,475 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection834: begin an election at term 941 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:01,491 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection834: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t941, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t941] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t941, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:01,491 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 941 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:01,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:01,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:01,535 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:01,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:01,535 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 941 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:01,536 [Thread-4359] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127725ms datanode_2 | 2019-10-15 03:16:28,328 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 735 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:01,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:01,549 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection835: begin an election at term 942 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:01,554 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection835: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t942, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:28,382 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_1 | 2019-10-15 03:17:01,575 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection835: begin an election at term 943 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,382 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:01,581 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection835: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t943, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t943] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t943, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:01,581 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 943 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,382 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 735 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:28,398 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection659: begin an election at term 736 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,400 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 727 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:01,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,400 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:01,581 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,400 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:01,748 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_2 | 2019-10-15 03:16:28,400 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:01,748 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,402 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection659 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:01,748 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 943 for changeToCandidate datanode_1 | 2019-10-15 03:17:01,748 [Thread-4365] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 127937ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:01,749 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:01,762 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: begin an election at term 944 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:01,777 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t944, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:01,791 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: begin an election at term 945 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:01,798 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t945, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:01,817 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: begin an election at term 946 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:01,826 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection836: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t946, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t946] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t946, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:01,826 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 946 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:01,826 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:01,827 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:02,023 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:02,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:02,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 946 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:02,023 [Thread-4373] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128212ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:02,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:02,038 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection837: begin an election at term 947 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:02,047 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection837: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t947, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t947] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t947, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:02,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 947 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:02,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:02,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:02,124 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_2 | 2019-10-15 03:16:28,425 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection659: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t736] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t736, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:02,124 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:02,124 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 947 for changeToCandidate datanode_1 | 2019-10-15 03:17:02,124 [Thread-4377] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128313ms datanode_1 | 2019-10-15 03:17:02,124 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:02,137 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection838: begin an election at term 948 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:02,145 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection838: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t948, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t948] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t948, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,425 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:02,145 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 948 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:02,145 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:02,145 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:02,256 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:02,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:02,256 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 948 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:02,256 [Thread-4381] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128445ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:02,256 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:02,268 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection839: begin an election at term 949 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:02,290 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection839: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t949, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t949] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t949, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:02,290 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 949 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:02,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:02,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:02,412 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:02,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:02,412 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 949 for changeToCandidate datanode_1 | 2019-10-15 03:17:02,412 [Thread-4385] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128601ms datanode_1 | 2019-10-15 03:17:02,412 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:02,426 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection840: begin an election at term 950 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:02,440 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection840: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t950, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:02,465 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection840: begin an election at term 951 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:28,425 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 736 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:02,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection840: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t951, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t951] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t951, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:02,482 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 951 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:02,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 728 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:02,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:02,562 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_2 | 2019-10-15 03:16:28,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:02,562 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,555 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:02,562 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 951 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,584 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 729 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:02,563 [Thread-4391] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128752ms datanode_2 | 2019-10-15 03:16:28,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:02,563 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:02,588 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection841: begin an election at term 952 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:02,614 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection841: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t952, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t952] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t952, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,584 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:02,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 952 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,601 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_1 | 2019-10-15 03:17:02,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:02,615 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,601 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 736 for changeToCandidate datanode_1 | 2019-10-15 03:17:02,778 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_2 | 2019-10-15 03:16:28,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:02,778 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,611 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection660: begin an election at term 737 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:02,778 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 952 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,614 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection660 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:02,778 [Thread-4395] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 128967ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:02,779 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:02,793 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection842: begin an election at term 953 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:02,808 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection842: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t953, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t953] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t953, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:02,808 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 953 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:02,808 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:02,808 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:02,970 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:02,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:02,971 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 953 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:02,971 [Thread-4399] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 129160ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:02,972 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:02,998 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection843: begin an election at term 954 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:03,028 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection843: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t954, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t954] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t954, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:03,028 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 954 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:03,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:03,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:03,161 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:03,161 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:03,161 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 954 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:03,161 [Thread-4403] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 129350ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:03,161 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:03,176 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection844: begin an election at term 955 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,626 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection660: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t737] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t737, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:03,194 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection844: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t955, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t955] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t955, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,626 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:03,194 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 955 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:03,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:03,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:03,376 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:03,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:03,376 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 955 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:03,377 [Thread-4407] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 129565ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:03,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:03,408 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection845: begin an election at term 956 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:03,436 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection845: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t956, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t956] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t956, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:03,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 956 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:03,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:03,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:03,607 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:03,607 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:03,607 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 956 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:03,607 [Thread-4411] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 129796ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:03,608 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:03,621 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection846: begin an election at term 957 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:03,647 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection846: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t957, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t957] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t957, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:03,647 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 957 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,626 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 737 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:03,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:03,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:03,833 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_2 | 2019-10-15 03:16:28,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:03,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,777 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 730 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:03,833 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 957 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,777 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:03,834 [Thread-4415] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130022ms datanode_2 | 2019-10-15 03:16:28,778 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:28,778 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:28,818 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_1 | 2019-10-15 03:17:03,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,818 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:03,848 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection847: begin an election at term 958 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,818 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 737 for changeToCandidate datanode_1 | 2019-10-15 03:17:03,855 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection847: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t958, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,818 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:03,876 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection847: begin an election at term 959 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,832 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection661: begin an election at term 738 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,835 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection661 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:03,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection847: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t959, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t959] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t959, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:03,901 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 959 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:03,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:03,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:04,000 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:04,001 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,001 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 959 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:04,001 [Thread-4421] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130190ms datanode_1 | 2019-10-15 03:17:04,001 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:04,015 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection848: begin an election at term 960 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,042 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection848: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t960, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t960] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t960, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:04,042 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 960 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:04,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:04,043 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:04,089 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:04,089 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:04,089 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 960 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:04,089 [Thread-4425] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130278ms datanode_1 | 2019-10-15 03:17:04,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,115 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection849: begin an election at term 961 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:04,166 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection849: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t961, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t961] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t961, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:04,166 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 961 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:04,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,850 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection661: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t738] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t738, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:04,272 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_2 | 2019-10-15 03:16:28,850 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:04,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 961 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,273 [Thread-4429] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130462ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:04,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:04,286 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection850: begin an election at term 962 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:04,294 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection850: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t962, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t962] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t962, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:04,294 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 962 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:04,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:04,294 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:04,366 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:04,367 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 962 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:04,367 [Thread-4433] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130556ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:04,367 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:04,381 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection851: begin an election at term 963 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:04,403 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection851: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t963, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t963] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t963, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:04,403 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 963 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,403 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:04,403 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,405 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:04,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,405 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 963 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:04,405 [Thread-4437] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130594ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:04,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,850 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 738 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:04,417 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection852: begin an election at term 964 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:04,433 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection852: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t964, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t964] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t964, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,931 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 731 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:04,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 964 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:28,931 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:28,931 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:04,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:28,931 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:04,461 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_2 | 2019-10-15 03:16:28,975 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_1 | 2019-10-15 03:17:04,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:28,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,461 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 964 for changeToCandidate datanode_2 | 2019-10-15 03:16:28,975 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 738 for changeToCandidate datanode_1 | 2019-10-15 03:17:04,461 [Thread-4441] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130650ms datanode_1 | 2019-10-15 03:17:04,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:28,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:04,495 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection853: begin an election at term 965 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,989 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection662: begin an election at term 739 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:04,511 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection853: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t965, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t965] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t965, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:28,992 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection662 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:04,511 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 965 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:04,511 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:04,634 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:04,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:04,634 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 965 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:04,634 [Thread-4445] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130823ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:04,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection854: begin an election at term 966 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:04,661 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection854: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t966, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t966] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t966, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:04,661 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 966 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:04,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:04,661 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:04,725 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:04,725 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,725 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 966 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:04,725 [Thread-4449] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 130914ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,725 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:04,739 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection855: begin an election at term 967 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:04,768 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection855: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t967, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t967] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t967, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:04,768 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 967 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:04,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:04,768 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:04,843 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_2 | 2019-10-15 03:16:29,004 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection662: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t739] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t739, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:04,843 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:04,843 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 967 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,004 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:04,844 [Thread-4453] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131032ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:04,844 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:04,857 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection856: begin an election at term 968 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:04,885 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection856: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t968, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t968] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t968, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:04,885 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 968 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:04,885 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:04,885 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:04,970 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:04,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:04,970 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 968 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:04,970 [Thread-4457] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131159ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:04,970 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:04,984 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection857: begin an election at term 969 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:05,013 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection857: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t969, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t969] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t969, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:05,013 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 969 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,013 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:05,014 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:05,046 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,046 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:05,046 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 969 for changeToCandidate datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:29,005 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 739 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:05,047 [Thread-4461] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131235ms datanode_2 | 2019-10-15 03:16:29,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:05,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,005 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:05,060 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection858: begin an election at term 970 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,077 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection858: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t970, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t970] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t970, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,077 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 970 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,045 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 732 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:05,077 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,045 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,077 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,045 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:05,086 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | 2019-10-15 03:16:29,045 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:05,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,086 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 970 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,093 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 733 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:05,087 [Thread-4465] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131275ms datanode_2 | 2019-10-15 03:16:29,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,093 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:05,100 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection859: begin an election at term 971 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,093 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:05,115 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection859: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t971, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t971] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t971, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,102 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_1 | 2019-10-15 03:17:05,115 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 971 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:05,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,102 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 739 for changeToCandidate datanode_1 | 2019-10-15 03:17:05,209 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:93ms, electionTimeout:93ms datanode_2 | 2019-10-15 03:16:29,102 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:05,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,119 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection663: begin an election at term 740 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,122 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection663 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:05,209 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 971 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:05,209 [Thread-4469] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131398ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:05,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:05,224 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection860: begin an election at term 972 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:05,237 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection860: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t972, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:05,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection860: begin an election at term 973 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection860: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t973, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t973] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t973, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:05,277 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 973 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:05,277 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:05,277 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:05,378 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:05,378 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:05,379 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 973 for changeToCandidate datanode_1 | 2019-10-15 03:17:05,379 [Thread-4475] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131568ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:05,379 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,392 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection861: begin an election at term 974 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:05,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection861: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t974, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t974] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t974, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,420 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 974 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:05,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:05,420 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:05,592 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_2 | 2019-10-15 03:16:29,135 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection663: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t740] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t740, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,135 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:05,592 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 974 for changeToCandidate datanode_1 | 2019-10-15 03:17:05,592 [Thread-4479] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131781ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:05,592 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:05,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection862: begin an election at term 975 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,619 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection862: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t975, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t975] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t975, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,619 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 975 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:05,619 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:05,619 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:05,784 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:05,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:05,785 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 975 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:05,785 [Thread-4483] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 131974ms datanode_1 | 2019-10-15 03:17:05,785 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,797 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection863: begin an election at term 976 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:05,802 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection863: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t976, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:05,811 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection863: begin an election at term 977 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:05,847 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection863: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t977, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t977] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t977, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,135 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 740 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:05,848 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 977 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:05,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:05,848 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,135 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:05,925 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_2 | 2019-10-15 03:16:29,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 734 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:05,925 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,925 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 977 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:05,926 [Thread-4489] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132114ms datanode_2 | 2019-10-15 03:16:29,316 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:05,926 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,323 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:187ms, electionTimeout:187ms datanode_1 | 2019-10-15 03:17:05,951 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection864: begin an election at term 978 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:05,969 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection864: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t978, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:05,980 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection864: begin an election at term 979 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,323 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 740 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,009 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection864: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t979, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t979] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t979, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:06,009 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 979 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:06,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:06,009 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,340 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection664: begin an election at term 741 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:06,026 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | 2019-10-15 03:16:29,344 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection664 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:06,026 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,026 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 979 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:06,026 [Thread-4495] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132215ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:06,026 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:06,040 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection865: begin an election at term 980 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:06,065 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection865: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t980, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t980] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t980, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:06,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 980 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:06,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:06,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:06,172 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:06,173 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 980 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:06,173 [Thread-4499] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132362ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:06,173 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:06,187 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection866: begin an election at term 981 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:06,214 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection866: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t981, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t981] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t981, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:06,215 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 981 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:06,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:06,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:06,275 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:06,275 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,356 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection664: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t741] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t741, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:06,275 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 981 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,276 [Thread-4503] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132464ms datanode_2 | 2019-10-15 03:16:29,357 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:06,276 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:06,289 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection867: begin an election at term 982 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,316 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection867: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t982, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t982] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t982, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:06,316 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 982 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:06,316 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:06,316 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:06,516 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_1 | 2019-10-15 03:17:06,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:06,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 982 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,516 [Thread-4507] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132705ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:06,530 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection868: begin an election at term 983 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:06,558 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection868: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t983, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t983] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t983, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:06,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 983 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:06,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:06,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:06,611 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:06,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:06,611 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 983 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,611 [Thread-4511] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132800ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:06,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:06,626 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection869: begin an election at term 984 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:06,654 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection869: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t984, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t984] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t984, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:06,654 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 984 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:06,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:06,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:06,728 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_1 | 2019-10-15 03:17:06,728 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,357 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 741 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:06,728 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 984 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,729 [Thread-4515] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 132917ms datanode_2 | 2019-10-15 03:16:29,357 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:06,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,357 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:06,745 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection870: begin an election at term 985 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,376 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:19ms, electionTimeout:19ms datanode_1 | 2019-10-15 03:17:06,762 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection870: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t985, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t985] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t985, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,376 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,376 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 741 for changeToCandidate datanode_1 | 2019-10-15 03:17:06,762 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 985 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,377 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:06,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,381 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection665: begin an election at term 742 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:06,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:06,901 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_2 | 2019-10-15 03:16:29,384 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection665 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:06,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,902 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 985 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:06,902 [Thread-4519] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133091ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:06,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:06,927 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection871: begin an election at term 986 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:06,955 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection871: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t986, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t986] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t986, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:06,955 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 986 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:06,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:06,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:07,112 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:07,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:07,112 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 986 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:07,112 [Thread-4523] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133301ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:07,112 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:07,149 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection872: begin an election at term 987 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:07,176 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection872: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t987, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t987] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t987, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 987 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:07,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:07,291 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:07,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:29,387 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection665: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t742] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t742, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:07,291 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 987 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,387 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:07,291 [Thread-4527] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133480ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:07,291 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:07,305 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection873: begin an election at term 988 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:07,333 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection873: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t988, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t988] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t988, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:07,333 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 988 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:07,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:07,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:07,528 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:07,528 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:07,528 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 988 for changeToCandidate datanode_1 | 2019-10-15 03:17:07,528 [Thread-4531] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133717ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:07,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:07,554 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection874: begin an election at term 989 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:07,571 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection874: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t989, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t989] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t989, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:07,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 989 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:07,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:07,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:07,598 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:07,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 989 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:07,599 [Thread-4535] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133787ms datanode_1 | 2019-10-15 03:17:07,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,612 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection875: begin an election at term 990 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:07,630 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection875: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t990, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t990] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t990, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:07,630 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 990 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:07,630 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,387 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 742 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:07,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:07,680 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_1 | 2019-10-15 03:17:07,680 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:07,680 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 990 for changeToCandidate datanode_1 | 2019-10-15 03:17:07,680 [Thread-4539] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133869ms datanode_2 | 2019-10-15 03:16:29,399 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_1 | 2019-10-15 03:17:07,680 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:07,693 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection876: begin an election at term 991 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,399 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 742 for changeToCandidate datanode_1 | 2019-10-15 03:17:07,715 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection876: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t991, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t991] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t991, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:07,715 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 991 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,405 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection666: begin an election at term 743 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:07,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,408 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection666 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:07,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:07,722 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:6ms, electionTimeout:6ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:07,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:07,722 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 991 for changeToCandidate datanode_1 | 2019-10-15 03:17:07,722 [Thread-4543] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133911ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:07,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:07,730 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection877: begin an election at term 992 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:07,757 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection877: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t992, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t992] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t992, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:07,757 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 992 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:07,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:07,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:07,765 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:7ms, electionTimeout:7ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:07,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:07,765 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 992 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:07,765 [Thread-4547] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 133954ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:07,766 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,779 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection878: begin an election at term 993 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:07,806 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection878: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t993, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t993] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t993, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:07,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 993 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:07,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:07,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:29,429 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection666: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t743] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t743, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:07,933 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | 2019-10-15 03:16:29,429 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:07,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:07,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 993 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:07,933 [Thread-4551] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134122ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:07,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:07,947 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection879: begin an election at term 994 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:07,974 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection879: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t994, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:08,000 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection879: begin an election at term 995 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:08,030 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection879: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t995, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t995] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t995, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:08,030 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 995 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:08,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:08,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:08,067 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:08,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:29,429 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 743 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:08,067 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 995 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,429 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:08,068 [Thread-4557] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134257ms datanode_2 | 2019-10-15 03:16:29,430 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:08,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 735 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:29,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:29,484 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:29,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_1 | 2019-10-15 03:17:08,081 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection880: begin an election at term 996 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:08,098 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection880: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t996, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t996] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t996, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 743 for changeToCandidate datanode_1 | 2019-10-15 03:17:08,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 996 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:08,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,507 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 736 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:29,507 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:08,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,507 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:08,164 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_2 | 2019-10-15 03:16:29,507 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:08,164 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,528 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection667: begin an election at term 744 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:08,164 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 996 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,532 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection667 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:08,164 [Thread-4561] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134353ms datanode_1 | 2019-10-15 03:17:08,164 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:08,177 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection881: begin an election at term 997 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:08,204 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection881: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t997, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t997] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t997, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:08,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 997 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:08,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:08,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:08,315 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:08,315 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:08,315 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 997 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:08,315 [Thread-4565] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134504ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:08,316 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:08,329 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection882: begin an election at term 998 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:08,357 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection882: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t998, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t998] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t998, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:08,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 998 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:08,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:08,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:08,386 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:08,386 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:08,386 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 998 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:08,386 [Thread-4569] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134575ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:08,387 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:08,400 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection883: begin an election at term 999 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:08,428 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection883: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t999, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t999] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t999, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,554 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection667: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t744] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t744, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:08,428 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 999 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,554 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:08,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:08,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:08,619 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:08,619 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:08,619 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 999 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:08,620 [Thread-4573] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 134808ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:08,620 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:08,645 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection884: begin an election at term 1000 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:08,673 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection884: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1000, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1000] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1000, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:08,673 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1000 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:08,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:08,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:08,854 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:180ms, electionTimeout:180ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:08,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:08,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1000 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:08,854 [Thread-4577] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135043ms datanode_1 | 2019-10-15 03:17:08,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:08,860 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection885: begin an election at term 1001 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:08,868 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection885: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1001, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1001] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1001, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:08,868 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1001 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:08,868 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:08,868 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 744 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:08,964 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | 2019-10-15 03:16:29,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:08,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:08,964 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1001 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,619 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_1 | 2019-10-15 03:17:08,964 [Thread-4581] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135153ms datanode_2 | 2019-10-15 03:16:29,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:08,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 744 for changeToCandidate datanode_1 | 2019-10-15 03:17:08,977 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection886: begin an election at term 1002 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:08,993 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection886: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1002, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1002] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1002, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,633 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection668: begin an election at term 745 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,636 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection668 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:08,993 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1002 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:08,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:08,993 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:09,055 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_1 | 2019-10-15 03:17:09,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:09,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1002 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:09,055 [Thread-4585] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135244ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:09,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:09,068 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection887: begin an election at term 1003 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:09,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection887: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1003, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1003] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1003, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:09,099 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1003 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:09,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:09,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:09,179 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:09,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:09,180 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1003 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:09,180 [Thread-4589] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135369ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:09,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:09,206 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection888: begin an election at term 1004 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:09,234 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection888: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1004, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1004] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1004, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:09,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1004 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:09,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:09,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:09,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:09,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,648 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection668: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t745] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t745, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:09,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1004 for changeToCandidate datanode_1 | 2019-10-15 03:17:09,271 [Thread-4593] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135459ms datanode_2 | 2019-10-15 03:16:29,648 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:09,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:09,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection889: begin an election at term 1005 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:09,313 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection889: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1005, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1005] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1005, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:09,313 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1005 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:09,313 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:09,313 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:09,380 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:09,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:09,381 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1005 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:09,381 [Thread-4597] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135570ms datanode_1 | 2019-10-15 03:17:09,381 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:09,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection890: begin an election at term 1006 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:09,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection890: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1006, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1006] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1006, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:09,403 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1006 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:09,403 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:09,403 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:09,481 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_1 | 2019-10-15 03:17:09,481 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:09,481 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1006 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:09,482 [Thread-4601] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135670ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:09,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:09,488 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection891: begin an election at term 1007 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:09,498 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection891: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1007, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1007] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1007, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:09,498 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1007 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,649 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 745 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:09,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:09,498 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:09,681 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_1 | 2019-10-15 03:17:09,681 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,665 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_1 | 2019-10-15 03:17:09,681 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1007 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:09,682 [Thread-4605] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 135870ms datanode_2 | 2019-10-15 03:16:29,665 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 745 for changeToCandidate datanode_1 | 2019-10-15 03:17:09,682 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:09,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection892: begin an election at term 1008 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,668 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 737 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:09,720 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection892: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1008, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:29,668 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:09,726 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection892: begin an election at term 1009 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,669 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection669: begin an election at term 746 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:09,753 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection892: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1009, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1009] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1009, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection669 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:09,753 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1009 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:09,753 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:09,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:09,903 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:09,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:09,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1009 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:09,903 [Thread-4611] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136092ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:09,904 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:09,925 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection893: begin an election at term 1010 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:09,964 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection893: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1010, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1010] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1010, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:09,965 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1010 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:09,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:09,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:09,985 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:09,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:09,985 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1010 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:09,985 [Thread-4615] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136174ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:09,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:10,001 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection894: begin an election at term 1011 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:10,028 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection894: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1011, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1011] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1011, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:10,028 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1011 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:10,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,676 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection669: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t746] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t746, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,029 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:10,055 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_2 | 2019-10-15 03:16:29,676 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:10,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:10,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1011 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:10,055 [Thread-4619] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136244ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:10,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:10,068 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection895: begin an election at term 1012 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:10,096 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection895: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1012, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1012] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1012, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:10,096 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1012 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:10,096 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:10,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:10,114 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:10,114 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:10,114 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1012 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:10,114 [Thread-4623] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136303ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:10,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:10,128 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection896: begin an election at term 1013 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:10,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection896: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1013, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1013] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1013, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:10,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1013 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:10,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:10,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:10,354 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:198ms, electionTimeout:198ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:10,355 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:10,355 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1013 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 746 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:10,355 [Thread-4627] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136544ms datanode_2 | 2019-10-15 03:16:29,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:10,355 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:10,379 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection897: begin an election at term 1014 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,790 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_1 | 2019-10-15 03:17:10,386 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection897: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1014, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1014] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1014, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:10,386 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1014 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,790 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 746 for changeToCandidate datanode_1 | 2019-10-15 03:17:10,386 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:10,386 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:10,396 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_1 | 2019-10-15 03:17:10,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:10,397 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1014 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:10,397 [Thread-4631] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136586ms datanode_2 | 2019-10-15 03:16:29,800 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection670: begin an election at term 747 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,397 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,801 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 738 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:10,412 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection898: begin an election at term 1015 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:10,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection898: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1015, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1015] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1015, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,801 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:10,441 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1015 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,801 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:10,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:29,803 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection670 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:10,441 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:10,566 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:125ms, electionTimeout:125ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:10,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:10,566 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1015 for changeToCandidate datanode_1 | 2019-10-15 03:17:10,567 [Thread-4635] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136755ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:10,567 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:10,593 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection899: begin an election at term 1016 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:10,621 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection899: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1016, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1016] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1016, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,621 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1016 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:10,621 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:10,621 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:10,664 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:10,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:10,664 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1016 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:10,664 [Thread-4639] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136853ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:10,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:10,684 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection900: begin an election at term 1017 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:10,712 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection900: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1017, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1017] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1017, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:10,712 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1017 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:10,712 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:10,712 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:10,751 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_2 | 2019-10-15 03:16:29,826 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection670: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t747] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t747, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,751 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,826 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:10,751 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1017 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:10,752 [Thread-4643] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 136940ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:10,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:10,767 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection901: begin an election at term 1018 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,794 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection901: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1018, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1018] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1018, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:10,794 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1018 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:10,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:10,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:10,981 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_1 | 2019-10-15 03:17:10,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:10,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1018 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:10,981 [Thread-4647] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137170ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:10,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:10,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection902: begin an election at term 1019 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:11,016 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection902: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1019, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1019] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1019, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:11,017 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1019 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:11,017 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:11,017 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:11,047 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:11,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:11,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1019 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:11,048 [Thread-4651] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137236ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:11,048 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:11,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection903: begin an election at term 1020 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:11,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection903: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1020, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1020] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1020, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:11,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1020 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:11,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:11,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:29,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 747 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:11,254 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_2 | 2019-10-15 03:16:29,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:11,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:29,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:11,254 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1020 for changeToCandidate datanode_2 | 2019-10-15 03:16:29,901 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 739 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:11,254 [Thread-4655] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137443ms datanode_2 | 2019-10-15 03:16:29,901 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:11,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:29,901 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:11,270 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection904: begin an election at term 1021 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,902 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:11,279 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection904: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1021, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,991 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:17:11,288 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection904: begin an election at term 1022 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:11,325 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection904: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1022, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1022] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1022, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:29,991 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 747 for changeToCandidate datanode_1 | 2019-10-15 03:17:11,325 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1022 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:29,992 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:11,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,005 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection671: begin an election at term 748 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:11,325 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:11,436 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_2 | 2019-10-15 03:16:30,009 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection671 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:11,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:11,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1022 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:11,436 [Thread-4661] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137625ms datanode_1 | 2019-10-15 03:17:11,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:11,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection905: begin an election at term 1023 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:11,479 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection905: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1023, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1023] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1023, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:11,479 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1023 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:11,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:11,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:11,585 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:17:11,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:11,585 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1023 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:11,586 [Thread-4665] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137774ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:11,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:11,599 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection906: begin an election at term 1024 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:11,627 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection906: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1024, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1024] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1024, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:11,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1024 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:11,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:11,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:11,639 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:11,640 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:11,640 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1024 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,021 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection671: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t748] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t748, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:11,640 [Thread-4669] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 137829ms datanode_2 | 2019-10-15 03:16:30,021 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:11,640 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:11,653 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection907: begin an election at term 1025 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:11,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection907: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1025, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1025] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1025, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:11,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1025 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:11,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:11,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:11,849 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:11,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:11,849 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1025 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:11,850 [Thread-4673] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138038ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:11,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:11,863 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection908: begin an election at term 1026 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:11,880 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection908: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1026, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1026] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1026, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:11,880 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1026 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:11,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:11,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:11,904 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:11,904 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:11,904 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1026 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:11,904 [Thread-4677] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138093ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:11,904 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:11,930 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection909: begin an election at term 1027 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:11,940 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection909: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1027, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,022 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 748 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:11,945 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection909: begin an election at term 1028 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,022 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:11,952 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection909: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1028, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1028] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1028, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,022 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:11,952 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1028 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,070 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 740 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:11,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:11,952 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:12,066 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_2 | 2019-10-15 03:16:30,070 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:12,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:12,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1028 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,185 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_1 | 2019-10-15 03:17:12,066 [Thread-4683] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138255ms datanode_2 | 2019-10-15 03:16:30,185 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:12,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,185 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 748 for changeToCandidate datanode_1 | 2019-10-15 03:17:12,080 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection910: begin an election at term 1029 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:12,084 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection910: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1029, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,185 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:12,109 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection910: begin an election at term 1030 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:12,149 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection910: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1030, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1030] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1030, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,210 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection672: begin an election at term 749 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:12,149 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1030 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,213 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection672 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:12,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:12,149 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:12,250 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:12,250 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:12,250 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1030 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:12,250 [Thread-4689] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138439ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:12,250 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:12,275 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection911: begin an election at term 1031 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:12,292 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection911: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1031, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1031] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1031, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:12,292 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1031 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:12,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:12,292 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:12,312 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:20ms, electionTimeout:20ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:12,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:12,312 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1031 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:12,313 [Thread-4693] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138501ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:12,313 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:12,327 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection912: begin an election at term 1032 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,225 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection672: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t749] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t749, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:12,349 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection912: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1032, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1032] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1032, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,225 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:12,349 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1032 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:12,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:12,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:12,474 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:12,474 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:12,474 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1032 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:12,474 [Thread-4697] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138663ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:12,474 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:12,488 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection913: begin an election at term 1033 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:12,517 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection913: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1033, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1033] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1033, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:12,517 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1033 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:12,517 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:12,517 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:12,644 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:12,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:12,645 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1033 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:12,645 [Thread-4701] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 138834ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:12,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:12,658 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection914: begin an election at term 1034 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:12,685 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection914: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1034, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1034] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1034, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:12,686 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1034 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,226 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 749 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:12,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:12,686 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:12,837 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | 2019-10-15 03:16:30,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:12,837 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,241 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 741 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:12,837 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1034 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:12,837 [Thread-4705] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139026ms datanode_2 | 2019-10-15 03:16:30,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:12,838 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,241 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:12,851 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection915: begin an election at term 1035 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,243 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_1 | 2019-10-15 03:17:12,867 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection915: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1035, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1035] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1035, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:12,867 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1035 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,243 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 749 for changeToCandidate datanode_1 | 2019-10-15 03:17:12,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:12,867 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,259 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection673: begin an election at term 750 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,033 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | 2019-10-15 03:16:30,263 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection673 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:13,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1035 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:13,034 [Thread-4709] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139222ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:13,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:13,048 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection916: begin an election at term 1036 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,050 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection916: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1036, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:13,075 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection916: begin an election at term 1037 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:13,103 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection916: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1037, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1037] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1037, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:13,103 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1037 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:13,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:13,191 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:13,191 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:13,191 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1037 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:13,191 [Thread-4715] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139380ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:13,191 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:13,207 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection917: begin an election at term 1038 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:13,215 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection917: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1038, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1038] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1038, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:13,215 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1038 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:13,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:13,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:13,356 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_1 | 2019-10-15 03:17:13,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:13,356 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1038 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:13,356 [Thread-4719] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139545ms datanode_2 | 2019-10-15 03:16:30,265 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection673: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t750] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t750, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,265 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:13,368 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection918: begin an election at term 1039 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:13,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection918: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1039, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1039] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1039, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:13,385 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1039 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:13,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:13,385 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:13,414 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,414 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:13,414 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1039 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:13,414 [Thread-4723] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139603ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:13,414 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:13,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection919: begin an election at term 1040 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,442 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection919: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1040, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1040] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1040, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:13,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1040 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:13,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:13,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:13,500 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:57ms, electionTimeout:57ms datanode_1 | 2019-10-15 03:17:13,500 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:13,500 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1040 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:13,500 [Thread-4727] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139689ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:13,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:13,514 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection920: begin an election at term 1041 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:13,541 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection920: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1041, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1041] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1041, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,542 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1041 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,265 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 750 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:13,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:13,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,265 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:13,582 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:40ms, electionTimeout:40ms datanode_2 | 2019-10-15 03:16:30,266 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 742 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:13,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:13,582 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1041 for changeToCandidate datanode_1 | 2019-10-15 03:17:13,582 [Thread-4731] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139771ms datanode_2 | 2019-10-15 03:16:30,266 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:13,583 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,266 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:30,266 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:13,596 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection921: begin an election at term 1042 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,390 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_1 | 2019-10-15 03:17:13,611 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection921: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1042, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1042] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1042, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:13,611 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1042 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,390 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 750 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,390 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:30,404 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection674: begin an election at term 751 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,408 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection674 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:13,611 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,656 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_1 | 2019-10-15 03:17:13,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:13,656 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1042 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:13,656 [Thread-4735] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139845ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:13,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:13,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection922: begin an election at term 1043 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:13,710 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection922: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1043, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1043] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1043, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,710 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1043 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:13,710 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:13,710 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:13,805 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:13,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:13,805 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1043 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:13,805 [Thread-4739] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 139994ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:13,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:13,819 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection923: begin an election at term 1044 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection923: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1044, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1044] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1044, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:13,846 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1044 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:13,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:13,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:13,901 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:54ms, electionTimeout:54ms datanode_1 | 2019-10-15 03:17:13,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,421 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection674: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t751] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t751, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:13,901 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1044 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,421 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:13,901 [Thread-4743] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140090ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:13,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:13,924 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection924: begin an election at term 1045 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:13,956 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection924: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1045, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1045] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1045, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:13,956 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1045 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:13,957 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:13,957 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:14,033 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:14,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:14,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1045 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:14,034 [Thread-4747] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140222ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:14,034 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:14,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection925: begin an election at term 1046 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:14,054 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection925: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1046, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1046] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1046, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:14,054 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1046 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:14,054 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:14,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:14,228 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:14,228 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:14,228 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1046 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:14,229 [Thread-4751] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140417ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:14,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,421 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 751 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:14,254 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection926: begin an election at term 1047 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:14,270 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection926: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1047, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1047] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1047, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:14,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1047 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:14,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,421 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:14,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,464 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 743 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:14,332 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_2 | 2019-10-15 03:16:30,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:14,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,464 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:14,332 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1047 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,494 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_1 | 2019-10-15 03:17:14,332 [Thread-4755] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140521ms datanode_2 | 2019-10-15 03:16:30,495 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:14,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,495 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 751 for changeToCandidate datanode_1 | 2019-10-15 03:17:14,346 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection927: begin an election at term 1048 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:14,363 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection927: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1048, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1048] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1048, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,495 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:14,363 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1048 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,511 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection675: begin an election at term 752 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:14,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,514 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection675 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:14,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:14,534 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:14,534 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:14,534 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1048 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:14,534 [Thread-4759] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140723ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:14,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:14,553 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection928: begin an election at term 1049 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:14,568 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection928: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1049, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:14,581 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection928: begin an election at term 1050 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:14,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection928: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1050, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1050] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1050, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:14,616 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1050 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:14,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:14,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:14,683 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:14,683 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:14,683 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1050 for changeToCandidate datanode_1 | 2019-10-15 03:17:14,684 [Thread-4765] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 140872ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:14,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:14,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection929: begin an election at term 1051 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:14,717 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection929: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1051, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1051] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1051, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,526 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection675: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t752] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t752, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:14,717 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1051 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,526 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:14,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:14,717 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:14,870 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:14,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:14,870 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1051 for changeToCandidate datanode_1 | 2019-10-15 03:17:14,870 [Thread-4769] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141059ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:14,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:14,884 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection930: begin an election at term 1052 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:14,903 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection930: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1052, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1052] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1052, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:14,903 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1052 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:14,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:14,903 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:15,078 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:15,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:15,078 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1052 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:15,079 [Thread-4773] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141267ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,079 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:15,093 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection931: begin an election at term 1053 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,123 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection931: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1053, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1053] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1053, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:15,123 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1053 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:15,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:15,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,526 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 752 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:15,178 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_1 | 2019-10-15 03:17:15,178 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:15,178 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1053 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,526 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:15,179 [Thread-4777] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141368ms datanode_2 | 2019-10-15 03:16:30,542 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_1 | 2019-10-15 03:17:15,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:15,193 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection932: begin an election at term 1054 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,542 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 752 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,542 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:15,223 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection932: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1054, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1054] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1054, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,555 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection676: begin an election at term 753 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:15,223 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1054 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,558 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection676 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:15,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:15,223 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:15,273 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:15,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:15,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1054 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:15,274 [Thread-4781] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141462ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:15,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:15,298 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection933: begin an election at term 1055 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:15,326 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection933: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1055, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1055] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1055, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:15,326 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1055 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:15,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:15,326 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:15,460 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:15,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:15,461 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1055 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:15,461 [Thread-4785] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141650ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:15,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:15,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection934: begin an election at term 1056 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,515 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection934: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1056, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1056] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1056, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:15,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1056 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:15,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:15,684 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:15,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,582 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection676: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t753] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t753, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:15,684 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1056 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,582 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:15,685 [Thread-4789] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 141873ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:15,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:15,710 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection935: begin an election at term 1057 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:15,726 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection935: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1057, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1057] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1057, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:15,726 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1057 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:15,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:15,726 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:15,863 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:15,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:15,863 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1057 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:15,863 [Thread-4793] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142052ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:15,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:15,877 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection936: begin an election at term 1058 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:15,894 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection936: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1058, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1058] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1058, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,894 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1058 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:15,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:15,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:15,938 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:43ms, electionTimeout:43ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:15,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:15,938 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1058 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:15,938 [Thread-4797] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142127ms datanode_2 | 2019-10-15 03:16:30,582 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 753 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:15,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:15,953 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection937: begin an election at term 1059 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:15,988 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection937: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1059, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1059] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1059, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,619 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_1 | 2019-10-15 03:17:15,988 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1059 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:15,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:15,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 753 for changeToCandidate datanode_1 | 2019-10-15 03:17:16,022 [Thread-4801] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142211ms datanode_2 | 2019-10-15 03:16:30,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:16,022 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_2 | 2019-10-15 03:16:30,632 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection677: begin an election at term 754 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:16,022 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,635 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection677 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:16,022 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1059 for changeToCandidate datanode_1 | 2019-10-15 03:17:16,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:16,046 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection938: begin an election at term 1060 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:16,063 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection938: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1060, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1060] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1060, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:16,063 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1060 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:16,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:16,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:16,251 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:188ms, electionTimeout:188ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:16,251 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:16,252 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1060 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:16,252 [Thread-4805] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142441ms datanode_1 | 2019-10-15 03:17:16,252 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:16,277 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection939: begin an election at term 1061 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:16,312 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection939: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1061, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1061] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1061, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:16,312 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1061 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:16,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:16,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:16,433 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:16,433 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:16,433 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1061 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:16,434 [Thread-4809] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142622ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:16,434 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,646 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection677: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t754] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t754, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:16,467 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection940: begin an election at term 1062 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:16,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection940: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1062, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1062] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1062, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,646 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:16,496 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1062 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:16,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:16,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:16,636 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:16,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:16,636 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1062 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:16,637 [Thread-4813] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 142825ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:16,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:16,651 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection941: begin an election at term 1063 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:16,668 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection941: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1063, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1063] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1063, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:16,668 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1063 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:16,668 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:16,668 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:16,831 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:16,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:16,831 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1063 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:16,831 [Thread-4817] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143020ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:16,831 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:16,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection942: begin an election at term 1064 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:16,873 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection942: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1064, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1064] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1064, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:16,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1064 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,647 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 754 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:16,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:16,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:17,032 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | 2019-10-15 03:16:30,684 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 744 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:17,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,684 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,684 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:17,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1064 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,684 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:17,033 [Thread-4821] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143222ms datanode_1 | 2019-10-15 03:17:17,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,734 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_1 | 2019-10-15 03:17:17,058 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection943: begin an election at term 1065 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 754 for changeToCandidate datanode_1 | 2019-10-15 03:17:17,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection943: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1065, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1065] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1065, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:17,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1065 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,758 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection678: begin an election at term 755 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:17,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,761 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection678 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:17,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,141 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_1 | 2019-10-15 03:17:17,141 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:17,141 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1065 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:17,141 [Thread-4825] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143330ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:17,142 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:17,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection944: begin an election at term 1066 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:17,171 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection944: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1066, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1066] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1066, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:17,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1066 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:17,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:17,334 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:17,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:17,334 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1066 for changeToCandidate datanode_1 | 2019-10-15 03:17:17,335 [Thread-4829] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143523ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:17,335 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:17,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection945: begin an election at term 1067 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:17,366 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection945: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1067, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1067] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1067, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:17,366 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1067 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:17,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:17,366 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:17,551 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:184ms, electionTimeout:184ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:17,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:17,551 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1067 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,764 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection678: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t755] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t755, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:17,551 [Thread-4833] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143740ms datanode_2 | 2019-10-15 03:16:30,764 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:17,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,565 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection946: begin an election at term 1068 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:17,573 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection946: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1068, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:17,589 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection946: begin an election at term 1069 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:17,618 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection946: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1069, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1069] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1069, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:17,618 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1069 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:17,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:17,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:17,672 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:17,672 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1069 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:17,672 [Thread-4839] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143861ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:17,672 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:17,686 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection947: begin an election at term 1070 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:17,701 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection947: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1070, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1070] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1070, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:17,701 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1070 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:17,701 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:17,701 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:17,723 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:17,723 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:17,723 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1070 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:17,723 [Thread-4843] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143912ms datanode_2 | 2019-10-15 03:16:30,765 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 755 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:17,723 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:17,736 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection948: begin an election at term 1071 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,765 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:17,764 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection948: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1071, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1071] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1071, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,773 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:16:30,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:17,764 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1071 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,773 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 755 for changeToCandidate datanode_1 | 2019-10-15 03:17:17,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:17,764 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,777 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection679: begin an election at term 756 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:17,799 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_2 | 2019-10-15 03:16:30,779 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection679 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:17,799 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:17,799 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1071 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:17,799 [Thread-4847] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 143988ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:17,799 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection949: begin an election at term 1072 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:17,812 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection949: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1072, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1072] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1072, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:17,812 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1072 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:17,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:17,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:17,834 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:17,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:17,834 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1072 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:17,834 [Thread-4851] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144023ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:17,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:17,843 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection950: begin an election at term 1073 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:17,860 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection950: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1073, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,797 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection679: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t756] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t756, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:17,874 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection950: begin an election at term 1074 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,797 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:17,895 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection950: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1074, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1074] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1074, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:17,895 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1074 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:17,895 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:17,895 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:18,010 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:18,010 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:18,010 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1074 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:18,010 [Thread-4857] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144199ms datanode_1 | 2019-10-15 03:17:18,010 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:18,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection951: begin an election at term 1075 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:18,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection951: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1075, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1075] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1075, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:18,040 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1075 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:18,040 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:18,040 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:18,152 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:18,152 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:18,152 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1075 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:18,153 [Thread-4861] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144342ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:18,153 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:18,177 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection952: begin an election at term 1076 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:18,194 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection952: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1076, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1076] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1076, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:18,194 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1076 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:18,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:18,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,797 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 756 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:18,307 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_2 | 2019-10-15 03:16:30,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:18,307 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,797 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:18,307 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1076 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,826 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 745 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:18,307 [Thread-4865] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144496ms datanode_2 | 2019-10-15 03:16:30,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:18,307 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,826 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:18,321 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection953: begin an election at term 1077 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,826 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:18,349 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection953: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1077, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1077] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1077, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,897 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_1 | 2019-10-15 03:17:18,349 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1077 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,897 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:18,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,897 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 756 for changeToCandidate datanode_1 | 2019-10-15 03:17:18,349 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,897 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:18,401 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | 2019-10-15 03:16:30,909 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection680: begin an election at term 757 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:18,401 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,913 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection680 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:18,401 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1077 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:18,401 [Thread-4869] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144590ms datanode_1 | 2019-10-15 03:17:18,401 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:18,413 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection954: begin an election at term 1078 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:18,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection954: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1078, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1078] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1078, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:18,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1078 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:18,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:18,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:18,537 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:17:18,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:18,537 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1078 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:18,537 [Thread-4873] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144726ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:18,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:18,551 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection955: begin an election at term 1079 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:18,579 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection955: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1079, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1079] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1079, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:18,579 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1079 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:18,579 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:18,579 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:18,616 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:18,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:18,616 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1079 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:18,616 [Thread-4877] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 144805ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:18,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:18,640 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection956: begin an election at term 1080 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:18,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection956: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1080, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:18,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection956: begin an election at term 1081 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:18,675 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection956: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1081, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1081] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1081, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,927 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection680: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t757] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t757, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:18,675 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1081 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:18,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:30,927 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:18,675 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:18,834 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:18,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:18,834 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1081 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:18,834 [Thread-4883] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145023ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:18,834 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:18,848 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection957: begin an election at term 1082 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:18,875 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection957: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1082, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1082] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1082, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:18,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1082 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:18,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:18,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:19,031 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_1 | 2019-10-15 03:17:19,031 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:19,031 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1082 for changeToCandidate datanode_1 | 2019-10-15 03:17:19,031 [Thread-4887] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145220ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:19,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:19,057 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection958: begin an election at term 1083 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:19,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection958: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1083, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1083] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1083, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:19,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1083 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:19,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,220 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:19,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:19,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1083 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 757 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:19,220 [Thread-4891] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145409ms datanode_2 | 2019-10-15 03:16:30,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:19,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:19,233 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection959: begin an election at term 1084 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:19,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection959: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1084, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1084] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1084, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:19,261 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1084 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:19,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:19,261 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:19,423 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:161ms, electionTimeout:161ms datanode_2 | 2019-10-15 03:16:30,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_1 | 2019-10-15 03:17:19,423 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:30,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:19,423 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1084 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 757 for changeToCandidate datanode_1 | 2019-10-15 03:17:19,423 [Thread-4895] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145612ms datanode_2 | 2019-10-15 03:16:30,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:19,424 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,957 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection681: begin an election at term 758 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:19,449 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection960: begin an election at term 1085 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,960 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection681 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:19,487 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection960: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1085, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1085] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1085, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:19,487 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1085 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:19,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:19,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:19,535 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_1 | 2019-10-15 03:17:19,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:19,535 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1085 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:19,536 [Thread-4899] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145724ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:19,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:19,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection961: begin an election at term 1086 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:19,589 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection961: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1086, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1086] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1086, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:19,589 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1086 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:19,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:19,589 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:19,618 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:19,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,618 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1086 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:19,618 [Thread-4903] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145807ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:19,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection962: begin an election at term 1087 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:19,658 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection962: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1087, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1087] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1087, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:30,963 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection681: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t758] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t758, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:19,658 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1087 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,963 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:19,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:19,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:19,754 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_1 | 2019-10-15 03:17:19,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:19,754 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1087 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:19,755 [Thread-4907] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 145944ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:19,756 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:19,770 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection963: begin an election at term 1088 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:19,797 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection963: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1088, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1088] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1088, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:19,797 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1088 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:19,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:19,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:19,865 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:67ms, electionTimeout:67ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:19,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:19,865 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1088 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:19,865 [Thread-4911] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146054ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:19,865 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,879 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection964: begin an election at term 1089 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:19,907 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection964: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1089, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1089] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1089, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:19,907 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1089 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:19,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:30,963 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 758 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:30,963 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:19,907 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:30,964 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:20,085 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | 2019-10-15 03:16:30,983 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 746 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:20,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,085 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1089 for changeToCandidate datanode_2 | 2019-10-15 03:16:30,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,085 [Thread-4915] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146274ms datanode_2 | 2019-10-15 03:16:30,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:20,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:30,983 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:31,080 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_2 | 2019-10-15 03:16:31,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,110 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection965: begin an election at term 1090 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,081 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 758 for changeToCandidate datanode_1 | 2019-10-15 03:17:20,128 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection965: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1090, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:20,139 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection965: begin an election at term 1091 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,125 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection682: begin an election at term 759 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:20,154 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection965: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1091, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1091] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1091, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,128 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 747 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:20,154 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1091 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:20,154 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,128 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,154 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,128 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:20,259 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | 2019-10-15 03:16:31,129 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:20,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,132 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection682 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:20,259 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1091 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:20,260 [Thread-4921] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146448ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:20,260 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:20,273 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection966: begin an election at term 1092 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:20,301 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection966: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1092, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1092] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1092, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:20,301 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1092 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:20,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:20,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:20,376 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:20,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,376 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1092 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:20,376 [Thread-4925] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146565ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:20,376 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:20,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection967: begin an election at term 1093 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:20,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection967: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1093, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1093] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1093, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:20,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1093 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:20,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:20,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:20,560 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:20,560 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:20,560 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1093 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:20,561 [Thread-4929] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146749ms datanode_1 | 2019-10-15 03:17:20,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:20,600 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection968: begin an election at term 1094 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:20,631 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection968: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1094, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1094] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1094, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,144 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection682: Election REJECTED; received 1 response(s) [749b19c7-0772-44d2-8efd-0664e6aa0748<-5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c#0:FAIL-t759] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t759, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,144 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:20,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1094 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:20,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:20,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:20,655 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:20,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:20,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1094 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:20,655 [Thread-4933] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146844ms datanode_1 | 2019-10-15 03:17:20,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:20,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection969: begin an election at term 1095 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:20,698 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection969: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1095, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1095] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1095, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:20,698 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1095 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:20,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:20,698 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:20,786 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:20,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:20,786 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1095 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:20,786 [Thread-4937] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 146975ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:20,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:20,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection970: begin an election at term 1096 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:20,810 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection970: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1096, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1096] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1096, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:20,810 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1096 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:20,811 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,144 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 759 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:20,811 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:20,944 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | 2019-10-15 03:16:31,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:20,944 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,214 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_1 | 2019-10-15 03:17:20,944 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1096 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,214 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:20,945 [Thread-4941] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147133ms datanode_2 | 2019-10-15 03:16:31,215 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 759 for changeToCandidate datanode_1 | 2019-10-15 03:17:20,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:20,958 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection971: begin an election at term 1097 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,215 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:20,971 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection971: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1097, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,228 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection683: begin an election at term 760 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:20,985 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection971: begin an election at term 1098 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,231 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection683 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:21,001 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection971: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1098, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1098] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1098, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,001 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1098 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:21,001 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:21,001 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:21,064 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:62ms, electionTimeout:62ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:21,064 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:21,064 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1098 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,064 [Thread-4947] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147253ms datanode_1 | 2019-10-15 03:17:21,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:21,078 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection972: begin an election at term 1099 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:21,104 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection972: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1099, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1099] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1099, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:21,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1099 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:21,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:21,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:21,213 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:21,213 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:21,213 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1099 for changeToCandidate datanode_1 | 2019-10-15 03:17:21,214 [Thread-4951] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147403ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:21,214 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:21,228 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection973: begin an election at term 1100 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:21,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection973: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1100, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:21,256 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection973: begin an election at term 1101 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:21,285 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection973: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1101, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1101] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1101, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:21,285 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1101 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,258 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection683 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:21,285 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,285 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:21,462 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:176ms, electionTimeout:176ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:21,462 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:21,462 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1101 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:21,462 [Thread-4957] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147651ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:21,462 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:21,477 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection974: begin an election at term 1102 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:21,504 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection974: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1102, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1102] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1102, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:21,504 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1102 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:21,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:21,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:21,535 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_1 | 2019-10-15 03:17:21,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:21,535 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1102 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:21,535 [Thread-4961] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147724ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:21,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,259 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection683: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t760, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:21,548 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection975: begin an election at term 1103 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,259 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:21,564 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection975: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1103, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1103] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1103, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,564 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1103 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:21,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:21,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:21,709 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:21,709 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:21,709 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1103 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,709 [Thread-4965] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 147898ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:21,710 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:21,738 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection976: begin an election at term 1104 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:21,761 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection976: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1104, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1104] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1104, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:21,762 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1104 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:21,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:21,762 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:21,944 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:21,944 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:21,944 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1104 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:21,945 [Thread-4969] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148133ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:21,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:21,950 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection977: begin an election at term 1105 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,259 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:21,957 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection977: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1105, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1105] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1105, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:21,957 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1105 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:21,957 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:21,957 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:22,015 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:22,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:22,016 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1105 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:22,016 [Thread-4973] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148205ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:22,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:22,037 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection978: begin an election at term 1106 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:22,053 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection978: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1106, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1106] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1106, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:22,053 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1106 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:22,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:22,053 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:22,247 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:22,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:22,247 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1106 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:22,247 [Thread-4977] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148436ms datanode_1 | 2019-10-15 03:17:22,247 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:22,261 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection979: begin an election at term 1107 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:22,273 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection979: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1107, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1107] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1107, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:22,273 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1107 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:22,273 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:22,274 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:22,299 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:22,299 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,259 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 760 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:22,299 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1107 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:22,299 [Thread-4981] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148488ms datanode_1 | 2019-10-15 03:17:22,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:22,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection980: begin an election at term 1108 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,268 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 748 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:22,330 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection980: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1108, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1108] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1108, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,268 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:22,330 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1108 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,268 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:22,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,268 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:22,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:22,463 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | 2019-10-15 03:16:31,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_1 | 2019-10-15 03:17:22,463 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 760 for changeToCandidate datanode_1 | 2019-10-15 03:17:22,463 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1108 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:22,464 [Thread-4985] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148653ms datanode_1 | 2019-10-15 03:17:22,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,395 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection684: begin an election at term 761 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:22,478 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection981: begin an election at term 1109 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 749 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:22,494 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection981: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1109, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1109] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1109, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:22,494 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1109 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:22,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:22,494 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:22,646 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | 2019-10-15 03:16:31,399 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection684 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:22,646 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:22,646 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1109 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:22,646 [Thread-4989] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 148835ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:22,647 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:22,660 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection982: begin an election at term 1110 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:22,677 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection982: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1110, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1110] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1110, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:22,677 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1110 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:22,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:22,677 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:22,840 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:22,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:22,841 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1110 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:22,841 [Thread-4993] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149030ms datanode_1 | 2019-10-15 03:17:22,841 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:22,854 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection983: begin an election at term 1111 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:22,869 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection983: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1111, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1111] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1111, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:22,869 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1111 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:22,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:22,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,404 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection684 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:22,964 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:94ms, electionTimeout:94ms datanode_1 | 2019-10-15 03:17:22,964 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:22,964 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1111 for changeToCandidate datanode_1 | 2019-10-15 03:17:22,964 [Thread-4997] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149153ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:22,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:22,978 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection984: begin an election at term 1112 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:22,994 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection984: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1112, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1112] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1112, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:22,994 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1112 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:22,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:22,994 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:23,041 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:47ms, electionTimeout:47ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:23,041 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:23,042 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1112 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:23,042 [Thread-5001] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149231ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:23,042 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,055 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection985: begin an election at term 1113 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:23,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection985: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1113, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1113] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1113, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:23,073 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1113 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:23,073 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:23,073 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,131 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:23,131 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:23,132 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1113 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,405 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection684: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t761, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:23,132 [Thread-5005] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149321ms datanode_2 | 2019-10-15 03:16:31,405 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:23,132 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:23,145 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection986: begin an election at term 1114 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:23,162 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection986: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1114, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1114] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1114, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:23,162 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1114 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:23,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:23,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:23,267 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:23,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:23,267 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1114 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:23,267 [Thread-5009] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149456ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:23,267 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:23,281 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection987: begin an election at term 1115 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:23,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection987: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1115, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1115] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1115, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:23,296 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1115 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:23,296 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,297 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:23,470 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,470 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:23,470 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1115 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:23,470 [Thread-5013] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149659ms datanode_1 | 2019-10-15 03:17:23,471 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:23,493 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection988: begin an election at term 1116 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,405 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:23,520 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection988: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1116, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1116] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1116, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:23,520 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1116 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:23,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:23,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:23,655 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_1 | 2019-10-15 03:17:23,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:23,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1116 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:23,655 [Thread-5017] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 149844ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:23,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:23,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection989: begin an election at term 1117 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:23,684 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection989: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1117, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:23,708 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection989: begin an election at term 1118 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:23,734 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection989: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1118, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1118] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1118, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:23,734 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1118 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:23,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:23,734 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:23,924 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:189ms, electionTimeout:189ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:23,924 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1118 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:23,924 [Thread-5023] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 150113ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:23,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:23,948 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection990: begin an election at term 1119 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:23,967 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection990: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1119, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1119] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1119, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,405 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 761 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:23,967 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1119 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:23,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:23,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:24,043 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_1 | 2019-10-15 03:17:24,043 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:24,043 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1119 for changeToCandidate datanode_1 | 2019-10-15 03:17:24,044 [Thread-5027] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 150232ms datanode_2 | 2019-10-15 03:16:31,471 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:65ms, electionTimeout:65ms datanode_1 | 2019-10-15 03:17:24,044 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:24,058 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection991: begin an election at term 1120 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,471 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 761 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,471 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:24,074 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection991: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1120, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1120] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1120, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,477 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection685: begin an election at term 762 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:24,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1120 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection685 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:24,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:24,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:24,215 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:24,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:24,215 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1120 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:24,215 [Thread-5031] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 150404ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:24,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:24,239 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection992: begin an election at term 1121 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:24,254 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection992: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1121, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1121] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1121, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:24,254 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1121 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:24,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:24,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:24,436 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_1 | 2019-10-15 03:17:24,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:24,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1121 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:24,436 [Thread-5035] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 150625ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:24,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:24,462 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection993: begin an election at term 1122 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:24,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection993: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1122, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1122] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1122, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:24,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1122 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:24,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:24,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,483 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection685 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:24,664 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:177ms, electionTimeout:177ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:24,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:24,664 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1122 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:24,664 [Thread-5039] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 150853ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:24,664 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:24,690 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection994: begin an election at term 1123 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:24,706 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection994: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1123, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1123] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1123, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:24,706 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1123 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:24,706 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:24,706 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:24,849 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:24,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:24,849 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1123 for changeToCandidate datanode_1 | 2019-10-15 03:17:24,849 [Thread-5043] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151038ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:24,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:24,863 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection995: begin an election at term 1124 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:24,878 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection995: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1124, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1124] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1124, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:24,878 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1124 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:24,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:24,879 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:24,880 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:24,880 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,483 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection685: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t762, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:24,880 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1124 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,483 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:24,880 [Thread-5047] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151069ms datanode_1 | 2019-10-15 03:17:24,881 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:24,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection996: begin an election at term 1125 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:24,930 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection996: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1125] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1125, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:24,947 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection996: begin an election at term 1126 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:24,963 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection996: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1126, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1126] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1126, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:24,963 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1126 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:24,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:24,963 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:25,125 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:25,125 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:25,125 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1126 for changeToCandidate datanode_1 | 2019-10-15 03:17:25,126 [Thread-5053] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151314ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:25,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:25,190 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection997: begin an election at term 1127 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:25,207 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection997: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1127, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1127] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1127, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:25,207 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1127 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:25,207 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:25,207 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:25,299 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:25,299 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:25,300 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1127 for changeToCandidate datanode_1 | 2019-10-15 03:17:25,300 [Thread-5057] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151489ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:25,300 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:25,315 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection998: begin an election at term 1128 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,483 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:25,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection998: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1128, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1128] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1128, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:25,336 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1128 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:25,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:25,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:25,505 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:25,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:25,505 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1128 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:25,505 [Thread-5061] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151694ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:25,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:25,529 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection999: begin an election at term 1129 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:25,545 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection999: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1129, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1129] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1129, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:25,545 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1129 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:25,545 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:25,546 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:25,550 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:4ms, electionTimeout:4ms datanode_1 | 2019-10-15 03:17:25,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:25,550 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1129 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:25,550 [Thread-5065] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151739ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:25,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:25,563 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1000: begin an election at term 1130 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:25,572 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1000: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1130, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:25,578 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1000: begin an election at term 1131 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:25,598 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1000: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1131, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1131] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1131, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 762 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:31,555 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:71ms, electionTimeout:71ms datanode_1 | 2019-10-15 03:17:25,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1131 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 762 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:25,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,563 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection686: begin an election at term 763 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:25,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,567 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection686 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:25,745 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:147ms, electionTimeout:147ms datanode_1 | 2019-10-15 03:17:25,745 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:25,745 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1131 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:25,746 [Thread-5071] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151934ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:25,746 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:25,760 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1001: begin an election at term 1132 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:25,775 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1001: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1132, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1132] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1132, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:25,776 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1132 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:25,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:25,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:25,806 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:25,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:25,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1132 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:25,806 [Thread-5075] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 151995ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:25,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:25,820 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1002: begin an election at term 1133 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:25,836 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1002: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1133, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1133] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1133, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:25,836 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1133 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:25,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:25,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:26,031 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_2 | 2019-10-15 03:16:31,572 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection686 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:26,031 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:26,031 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1133 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:26,032 [Thread-5079] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152220ms datanode_1 | 2019-10-15 03:17:26,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:26,046 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1003: begin an election at term 1134 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:26,063 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1003: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1134, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1134] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1134, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:26,063 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1134 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:26,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:26,063 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:26,113 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:26,113 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:26,114 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1134 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:26,114 [Thread-5083] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152303ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:26,114 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:26,141 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1004: begin an election at term 1135 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:26,152 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1004: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1135, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:26,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1004: begin an election at term 1136 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,572 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection686: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t763, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,572 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:26,171 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1004: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1136, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1136] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1136, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:26,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1136 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:26,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:26,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:26,202 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:30ms, electionTimeout:30ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:26,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:26,202 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1136 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:26,202 [Thread-5089] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152391ms datanode_1 | 2019-10-15 03:17:26,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:26,216 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1005: begin an election at term 1137 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:26,232 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1005: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1137, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1137] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1137, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:26,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1137 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:26,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:26,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:26,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:26,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:26,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1137 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:26,289 [Thread-5093] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152478ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,572 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:26,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:26,302 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1006: begin an election at term 1138 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:26,309 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1006: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1138, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1138] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1138, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:26,309 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1138 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:26,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:26,309 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:26,488 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_1 | 2019-10-15 03:17:26,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:26,488 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1138 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:26,488 [Thread-5097] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152677ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:26,489 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:26,502 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1007: begin an election at term 1139 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:26,519 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1007: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1139, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1139] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1139, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:26,519 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1139 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:26,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:26,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:26,614 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | 2019-10-15 03:16:31,573 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 763 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:26,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:31,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 750 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:31,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:26,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1139 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:31,620 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:31,634 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 751 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:26,614 [Thread-5101] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152803ms datanode_2 | 2019-10-15 03:16:31,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:26,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:26,627 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1008: begin an election at term 1140 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,635 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:26,645 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1008: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1140, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1140] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1140, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,717 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_1 | 2019-10-15 03:17:26,645 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1140 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:26,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,717 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 763 for changeToCandidate datanode_1 | 2019-10-15 03:17:26,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:26,794 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | 2019-10-15 03:16:31,723 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection687: begin an election at term 764 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,727 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection687 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:26,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:26,794 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1140 for changeToCandidate datanode_1 | 2019-10-15 03:17:26,795 [Thread-5105] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 152983ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:26,795 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:26,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1009: begin an election at term 1141 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:26,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1009: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1141, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1141] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1141, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:26,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1141 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:26,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:26,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:26,958 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:26,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:26,958 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1141 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:26,958 [Thread-5109] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153147ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:26,958 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:26,972 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1010: begin an election at term 1142 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:26,991 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1010: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1142, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1142] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1142, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:26,991 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1142 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:26,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:26,991 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:27,055 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:27,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1142 for changeToCandidate datanode_1 | 2019-10-15 03:17:27,055 [Thread-5113] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153244ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:27,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,070 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1011: begin an election at term 1143 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:27,098 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1011: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1143, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1143] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1143, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:27,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1143 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:27,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,728 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection687 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:27,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,127 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:27,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:27,127 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1143 for changeToCandidate datanode_1 | 2019-10-15 03:17:27,127 [Thread-5117] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153316ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:27,128 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:27,141 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1012: begin an election at term 1144 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:27,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1012: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1144, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1144] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1144, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:27,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1144 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:27,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:27,331 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:27,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:27,332 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1144 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:27,332 [Thread-5121] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153521ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:27,332 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,346 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1013: begin an election at term 1145 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:27,372 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1013: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1145, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1145] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1145, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,372 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1145 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:27,372 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:27,372 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,728 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection687: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t764, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:27,400 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_2 | 2019-10-15 03:16:31,729 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:27,400 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,401 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1145 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:27,401 [Thread-5125] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153590ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:27,401 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:27,415 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1014: begin an election at term 1146 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:27,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1014: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1146, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1146] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1146, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1146 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:27,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:27,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:27,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:27,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1146 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:27,446 [Thread-5129] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153635ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:27,459 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1015: begin an election at term 1147 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:27,472 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1015: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1147, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,729 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:27,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1015: begin an election at term 1148 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,491 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1015: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1148, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1148] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1148, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:27,491 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1148 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:27,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:27,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:27,509 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:27,509 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:27,509 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1148 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:27,510 [Thread-5135] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153698ms datanode_1 | 2019-10-15 03:17:27,510 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,523 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1016: begin an election at term 1149 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:27,541 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1016: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1149, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1149] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1149, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:27,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1149 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:27,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:27,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:27,564 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:27,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:27,564 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1149 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,564 [Thread-5139] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153753ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:27,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:27,577 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1017: begin an election at term 1150 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:27,602 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1017: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1150, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1150] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1150, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:27,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1150 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,729 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 764 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:27,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,729 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:27,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,729 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:27,652 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_2 | 2019-10-15 03:16:31,730 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_1 | 2019-10-15 03:17:27,652 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,730 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 764 for changeToCandidate datanode_1 | 2019-10-15 03:17:27,652 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1150 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:31,755 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection688: begin an election at term 765 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:27,652 [Thread-5143] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 153841ms datanode_2 | 2019-10-15 03:16:31,757 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 752 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:27,652 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:27,666 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1018: begin an election at term 1151 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,757 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:27,681 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1018: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1151, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1151] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1151, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,757 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:27,681 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1151 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:27,682 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,759 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection688 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:27,682 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,849 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:27,849 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:27,849 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1151 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:27,849 [Thread-5147] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154038ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:27,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:27,875 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1019: begin an election at term 1152 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:27,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1019: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1152, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1152] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1152, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:27,901 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1152 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:27,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:27,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:28,005 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:28,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:28,006 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1152 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:28,006 [Thread-5151] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154195ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:28,006 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,020 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1020: begin an election at term 1153 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:28,035 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1020: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1153, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1153] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1153, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1153 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:28,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:28,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:28,146 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:110ms, electionTimeout:110ms datanode_2 | 2019-10-15 03:16:31,759 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection688 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:28,146 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,146 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1153 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:28,146 [Thread-5155] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154335ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:28,147 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:28,160 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1021: begin an election at term 1154 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:28,176 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1021: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1154, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1154] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1154, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:28,177 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1154 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:28,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:28,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:28,289 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:28,289 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:28,289 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1154 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:28,289 [Thread-5159] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154478ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,290 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:28,303 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1022: begin an election at term 1155 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,319 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1022: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1155, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1155] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1155, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:28,319 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1155 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:28,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,759 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection688: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t765, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:28,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,759 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:28,488 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:28,489 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1155 for changeToCandidate datanode_1 | 2019-10-15 03:17:28,489 [Thread-5163] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154678ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:28,489 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:28,502 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1023: begin an election at term 1156 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:28,519 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1023: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1156, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1156] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1156, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:28,519 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1156 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:28,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:28,519 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,568 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_1 | 2019-10-15 03:17:28,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:28,568 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1156 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:28,568 [Thread-5167] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154757ms datanode_1 | 2019-10-15 03:17:28,568 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:28,582 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1024: begin an election at term 1157 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:28,597 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1024: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1157, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1157] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1157, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:28,597 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1157 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:28,597 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,597 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:28,648 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:28,648 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1157 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:28,648 [Thread-5171] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 154837ms datanode_2 | 2019-10-15 03:16:31,760 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:28,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1025: begin an election at term 1158 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:28,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1025: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1158, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:28,706 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1025: begin an election at term 1159 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:28,731 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1025: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1159, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1159] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1159, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:28,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1159 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:28,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:28,836 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:28,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:28,837 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1159 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:28,837 [Thread-5177] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155026ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:28,837 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,850 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1026: begin an election at term 1160 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:28,866 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1026: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1160, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1160] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1160, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:28,866 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1160 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:28,866 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:28,866 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:28,888 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | 2019-10-15 03:16:31,760 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 765 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:28,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:28,888 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1160 for changeToCandidate datanode_1 | 2019-10-15 03:17:28,888 [Thread-5181] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155077ms datanode_2 | 2019-10-15 03:16:31,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:28,889 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,770 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_1 | 2019-10-15 03:17:28,902 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1027: begin an election at term 1161 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,770 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 765 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:28,917 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1027: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1161, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1161] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1161, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,781 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection689: begin an election at term 766 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:28,917 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1161 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,784 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection689 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:28,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:28,917 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:29,108 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:29,108 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:29,108 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1161 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:29,108 [Thread-5185] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155297ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:29,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:29,123 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1028: begin an election at term 1162 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:29,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1028: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1162, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1162] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1162, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:29,140 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1162 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:29,140 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:29,140 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:29,193 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:29,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:29,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1162 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:29,194 [Thread-5189] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155382ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:29,194 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:29,219 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1029: begin an election at term 1163 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:29,240 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1029: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1163, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1163] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1163, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,785 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection689 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:29,240 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1163 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:29,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:29,240 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:29,432 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_1 | 2019-10-15 03:17:29,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:29,432 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1163 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:29,432 [Thread-5193] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155621ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:29,432 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:29,482 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1030: begin an election at term 1164 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:29,494 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1030: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1164, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:29,498 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1030: begin an election at term 1165 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:29,512 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1030: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1165, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1165] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1165, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:29,513 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1165 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:29,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:29,513 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:29,551 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_1 | 2019-10-15 03:17:29,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,786 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection689: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t766, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,786 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:29,551 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1165 for changeToCandidate datanode_1 | 2019-10-15 03:17:29,551 [Thread-5199] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155740ms datanode_1 | 2019-10-15 03:17:29,551 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:29,565 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1031: begin an election at term 1166 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:29,580 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1031: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1166, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1166] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1166, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:29,580 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1166 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:29,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:29,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:29,662 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:81ms, electionTimeout:81ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:29,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:29,662 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1166 for changeToCandidate datanode_1 | 2019-10-15 03:17:29,662 [Thread-5203] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 155851ms datanode_1 | 2019-10-15 03:17:29,662 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:29,676 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1032: begin an election at term 1167 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:29,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1032: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1167, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1167] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1167, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:29,692 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1167 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:29,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:29,692 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:29,875 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:182ms, electionTimeout:182ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:29,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:29,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1167 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:29,875 [Thread-5207] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156064ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:29,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:29,889 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1033: begin an election at term 1168 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:29,916 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1033: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1168, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1168] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1168, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,786 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:29,916 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1168 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:29,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:29,916 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:30,025 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:30,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:30,026 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1168 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:30,026 [Thread-5211] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156214ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:30,026 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:30,039 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1034: begin an election at term 1169 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:30,060 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1034: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1169, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1169] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1169, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:30,060 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1169 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:30,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:30,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:30,163 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_1 | 2019-10-15 03:17:30,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:30,163 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1169 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,163 [Thread-5215] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156352ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:30,163 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,177 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1035: begin an election at term 1170 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:30,193 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1035: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1170, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1170] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1170, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:30,193 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1170 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:30,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,786 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 766 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:30,193 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:30,236 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | 2019-10-15 03:16:31,837 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 753 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:31,837 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:30,236 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,837 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:30,236 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1170 for changeToCandidate datanode_1 | 2019-10-15 03:17:30,236 [Thread-5219] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156425ms datanode_2 | 2019-10-15 03:16:31,837 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:30,236 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,868 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_1 | 2019-10-15 03:17:30,250 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1036: begin an election at term 1171 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:30,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1036: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1171, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1171] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1171, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,869 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 766 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,869 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:31,875 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection690: begin an election at term 767 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:30,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1171 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:31,878 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection690 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:30,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:30,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:30,353 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:30,353 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:30,353 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1171 for changeToCandidate datanode_1 | 2019-10-15 03:17:30,354 [Thread-5223] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156543ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:30,354 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:30,368 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1037: begin an election at term 1172 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:30,384 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1037: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1172, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1172] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1172, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:30,384 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1172 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:30,384 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:30,384 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:30,492 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:30,492 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:30,492 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1172 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:30,492 [Thread-5227] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156681ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:30,493 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,506 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1038: begin an election at term 1173 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:30,522 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1038: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1173, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1173] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1173, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,522 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1173 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,879 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection690 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:30,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:30,522 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:30,549 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:30,549 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:30,549 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1173 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:30,549 [Thread-5231] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156738ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:30,550 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:30,574 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1039: begin an election at term 1174 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:30,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1039: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1174, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1174] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1174, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:30,601 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1174 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:30,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:30,601 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:30,667 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:30,668 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:30,668 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1174 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,669 [Thread-5235] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156857ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:30,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,695 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1040: begin an election at term 1175 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:30,722 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1040: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1175, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1175] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1175, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:30,722 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1175 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:30,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,879 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection690: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t767, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:30,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:31,879 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:30,787 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:30,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:30,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1175 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:30,787 [Thread-5239] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 156976ms datanode_1 | 2019-10-15 03:17:30,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:30,801 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1041: begin an election at term 1176 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:30,818 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1041: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1176, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1176] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1176, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:30,818 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1176 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:30,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:30,818 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:30,977 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:159ms, electionTimeout:159ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:30,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:30,977 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1176 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,978 [Thread-5243] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157166ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:30,978 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:30,991 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1042: begin an election at term 1177 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:31,007 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1042: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1177, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1177] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1177, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:31,008 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1177 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:31,008 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:31,008 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:31,037 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:31,037 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,879 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:31,038 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1177 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:31,038 [Thread-5247] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157226ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:31,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:31,051 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1043: begin an election at term 1178 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:31,067 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1043: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1178, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1178] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1178, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:31,067 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1178 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:31,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:31,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:31,104 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:37ms, electionTimeout:37ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:31,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:31,105 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1178 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:31,105 [Thread-5251] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157294ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:31,105 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:31,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1044: begin an election at term 1179 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:31,136 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1044: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1179, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1179] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1179, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:31,136 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1179 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:31,136 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:31,136 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:31,248 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:31,880 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 767 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:31,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:31,880 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:31,880 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:31,248 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1179 for changeToCandidate datanode_2 | 2019-10-15 03:16:31,990 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 754 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:31,248 [Thread-5255] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157437ms datanode_1 | 2019-10-15 03:17:31,248 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:31,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:31,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: begin an election at term 1180 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:31,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1180, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:31,990 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:31,276 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: begin an election at term 1181 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,019 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 755 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:31,289 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1181, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:31,305 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: begin an election at term 1182 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,020 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:31,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1045: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1182, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1182] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1182, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,020 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:31,321 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1182 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_1 | 2019-10-15 03:17:31,321 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:31,321 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 767 for changeToCandidate datanode_1 | 2019-10-15 03:17:31,379 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_2 | 2019-10-15 03:16:32,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:31,379 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,075 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691: begin an election at term 768 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:31,379 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1182 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 756 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:32,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:31,379 [Thread-5264] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157568ms datanode_2 | 2019-10-15 03:16:32,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:32,076 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:31,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,078 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:31,423 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1046: begin an election at term 1183 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:31,453 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1046: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1183, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:31,490 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1046: begin an election at term 1184 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:31,512 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1046: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1184, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1184] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1184, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:31,512 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1184 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:31,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:31,512 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:31,655 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:31,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:31,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1184 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:31,655 [Thread-5270] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 157844ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:31,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:31,683 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1047: begin an election at term 1185 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:31,702 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1047: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1185, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1185] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1185, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:31,702 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1185 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:31,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:31,702 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:31,824 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:31,824 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,080 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t768, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:31,824 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1185 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,080 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:31,825 [Thread-5274] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158013ms datanode_1 | 2019-10-15 03:17:31,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:31,839 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1048: begin an election at term 1186 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:31,866 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1048: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1186, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1186] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1186, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:31,866 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1186 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:31,866 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:31,866 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:31,975 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:31,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:31,975 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1186 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:31,975 [Thread-5278] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158164ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:31,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:31,990 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1049: begin an election at term 1187 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:32,006 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1049: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1187, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1187] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1187, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:32,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1187 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:32,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:32,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,093 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:32,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:32,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1187 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:32,093 [Thread-5282] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158282ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:32,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,103 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691: begin an election at term 769 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,106 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:32,107 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1050: begin an election at term 1188 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:32,123 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1050: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1188, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1188] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1188, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:32,123 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1188 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:32,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:32,123 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:32,138 [Command processor thread] ERROR - Container #4 does not exist in datanode. Container close failed. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:32,138 [Command processor thread] ERROR - Container #5 does not exist in datanode. Container close failed. datanode_1 | 2019-10-15 03:17:32,232 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:32,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:32,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1188 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:32,232 [Thread-5286] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158421ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:32,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:32,246 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1051: begin an election at term 1189 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,263 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1051: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1189, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1189] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1189, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:32,263 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1189 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:32,263 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:32,286 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:23ms, electionTimeout:23ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:32,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,107 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:32,287 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1189 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:32,287 [Thread-5290] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158476ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:32,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:32,302 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1052: begin an election at term 1190 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:32,317 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1052: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1190, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1190] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1190, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:32,317 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1190 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:32,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:32,317 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:32,473 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:32,473 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:32,473 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1190 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:32,473 [Thread-5294] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158662ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:32,473 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,487 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1053: begin an election at term 1191 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:32,503 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1053: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1191, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1191] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1191, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:32,503 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1191 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:32,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:32,503 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:32,656 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:32,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,107 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection691: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t769, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,107 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:32,656 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1191 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:32,656 [Thread-5298] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158845ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:32,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:32,669 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1054: begin an election at term 1192 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:32,694 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1054: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1192, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1192] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1192, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:32,694 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1192 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:32,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:32,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:32,758 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:32,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:32,758 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1192 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:32,758 [Thread-5302] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 158947ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:32,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:32,772 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1055: begin an election at term 1193 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,788 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1055: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1193, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1193] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1193, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:32,788 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1193 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:32,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:32,788 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:32,938 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:32,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,108 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:32,938 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1193 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:32,939 [Thread-5306] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 159127ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:32,939 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:32,952 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1056: begin an election at term 1194 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:32,977 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1056: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1194, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1194] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1194, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:32,977 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1194 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:32,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:32,977 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:33,108 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:33,108 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:33,108 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1194 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:33,108 [Thread-5310] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 159297ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:33,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:33,122 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1057: begin an election at term 1195 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:33,151 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1057: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1195, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1195] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1195, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:33,151 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1195 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:33,151 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:33,151 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:33,196 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:33,196 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:33,196 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1195 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 769 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:33,197 [Thread-5314] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 159385ms datanode_2 | 2019-10-15 03:16:32,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:33,197 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:33,210 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1058: begin an election at term 1196 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,166 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 757 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:33,231 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1058: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1196, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1196] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1196, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,167 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:32,167 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:33,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1196 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,210 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_2 | 2019-10-15 03:16:32,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:33,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,211 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 769 for changeToCandidate datanode_1 | 2019-10-15 03:17:33,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,211 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:33,428 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | 2019-10-15 03:16:32,224 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection692: begin an election at term 770 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:33,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,228 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection692 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:33,428 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1196 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:33,428 [Thread-5318] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 159617ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:33,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:33,442 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1059: begin an election at term 1197 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:33,448 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1059: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1197, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1197] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1197, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:33,448 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1197 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:33,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:33,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:33,613 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:33,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:33,613 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1197 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:33,613 [Thread-5322] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 159802ms datanode_1 | 2019-10-15 03:17:33,613 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:33,627 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1060: begin an election at term 1198 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:33,655 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1060: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1198, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1198] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1198, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:33,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1198 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:33,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:33,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:33,823 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:167ms, electionTimeout:167ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:33,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:33,823 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1198 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:33,823 [Thread-5326] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160012ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:33,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,229 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection692 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:33,849 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1061: begin an election at term 1199 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:33,876 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1061: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1199, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1199] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1199, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:33,876 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1199 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:33,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:33,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:33,899 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:33,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:33,899 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1199 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:33,899 [Thread-5330] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160088ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:33,899 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:33,912 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1062: begin an election at term 1200 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:33,920 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1062: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1200, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1200] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1200, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:33,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1200 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:33,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:33,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:34,044 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:34,044 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:34,044 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1200 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:34,045 [Thread-5334] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160233ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:34,045 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,229 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection692: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t770, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,229 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:34,069 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1063: begin an election at term 1201 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,097 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1063: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1201, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1201] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1201, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:34,097 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1201 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:34,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:34,097 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,210 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:34,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:34,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1201 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:34,210 [Thread-5338] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160399ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:34,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:34,234 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1064: begin an election at term 1202 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:34,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1064: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1202, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1202] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1202, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:34,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1202 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:34,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:34,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:34,405 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:34,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:34,405 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1202 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:34,405 [Thread-5342] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160594ms datanode_2 | 2019-10-15 03:16:32,230 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:34,405 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1065: begin an election at term 1203 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:34,458 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1065: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1203, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1203] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1203, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:34,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1203 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:34,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:34,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:34,509 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_1 | 2019-10-15 03:17:34,509 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:34,509 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1203 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:34,509 [Thread-5346] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160698ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:34,509 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,531 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1066: begin an election at term 1204 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:34,533 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1066: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1204, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:34,558 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1066: begin an election at term 1205 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:34,585 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1066: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1205, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1205] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1205, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:34,585 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1205 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:34,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:34,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:34,645 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:34,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:34,645 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1205 for changeToCandidate datanode_1 | 2019-10-15 03:17:34,645 [Thread-5352] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160834ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:34,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:34,670 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1067: begin an election at term 1206 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:34,686 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1067: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1206, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1206] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1206, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:34,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1206 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:34,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:34,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:34,739 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_1 | 2019-10-15 03:17:34,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:34,739 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1206 for changeToCandidate datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,230 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 770 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:34,739 [Thread-5356] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160928ms datanode_2 | 2019-10-15 03:16:32,230 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:34,740 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,274 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:43ms, electionTimeout:43ms datanode_2 | 2019-10-15 03:16:32,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:34,753 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1068: begin an election at term 1207 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 770 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:34,761 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1068: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1207, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1207] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1207, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,287 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection693: begin an election at term 771 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection693 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:34,761 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1207 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:34,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:34,786 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:24ms, electionTimeout:24ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:34,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:34,786 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1207 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:34,786 [Thread-5360] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 160975ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:34,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:34,802 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1069: begin an election at term 1208 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:34,812 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1069: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1208, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1208] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1208, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:34,812 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1208 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:34,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:34,813 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:35,003 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:190ms, electionTimeout:190ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:35,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:35,003 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1208 for changeToCandidate datanode_1 | 2019-10-15 03:17:35,003 [Thread-5364] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161192ms datanode_1 | 2019-10-15 03:17:35,003 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:35,029 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1070: begin an election at term 1209 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:35,057 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1070: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1209, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1209] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1209, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:35,057 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1209 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection693 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:35,057 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,057 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:35,177 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:35,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:35,177 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1209 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:35,177 [Thread-5368] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161366ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:35,178 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,192 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1071: begin an election at term 1210 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:35,216 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1071: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1210, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1210] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1210, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:35,216 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1210 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:35,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:35,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:35,376 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:160ms, electionTimeout:160ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:35,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:35,377 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1210 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:35,377 [Thread-5372] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161566ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:35,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:35,391 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1072: begin an election at term 1211 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:35,399 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1072: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1211, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:35,417 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1072: begin an election at term 1212 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:35,434 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1072: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1212, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1212] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1212, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,292 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection693: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t771, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:35,434 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1212 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:35,434 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,292 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:35,434 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,586 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:35,586 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:35,586 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1212 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:35,587 [Thread-5378] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161775ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:35,587 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:35,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1073: begin an election at term 1213 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,622 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1073: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1213, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1213] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1213, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:35,622 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1213 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:35,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:35,622 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:35,737 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:35,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:35,737 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1213 for changeToCandidate datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,292 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:35,737 [Thread-5382] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161926ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:35,751 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1074: begin an election at term 1214 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:35,780 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1074: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1214, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1214] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1214, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:35,780 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1214 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:35,780 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:35,780 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:35,794 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:35,794 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:35,795 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1214 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:35,795 [Thread-5386] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 161983ms datanode_1 | 2019-10-15 03:17:35,795 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:35,809 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1075: begin an election at term 1215 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:35,825 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1075: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1215, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1215] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1215, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:35,825 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1215 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:35,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:35,825 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:35,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:35,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:35,842 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1215 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:35,842 [Thread-5390] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162031ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:35,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:35,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1076: begin an election at term 1216 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:35,876 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1076: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1216, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1216] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1216, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:35,876 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1216 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:35,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:35,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:35,989 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_2 | 2019-10-15 03:16:32,292 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 771 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:35,989 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:35,990 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1216 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:35,990 [Thread-5394] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162178ms datanode_2 | 2019-10-15 03:16:32,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:35,990 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,300 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_1 | 2019-10-15 03:17:36,004 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1077: begin an election at term 1217 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,300 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 771 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,301 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:36,033 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1077: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1217, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1217] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1217, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,314 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection694: begin an election at term 772 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:36,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1217 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,317 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection694 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:36,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:36,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:36,129 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:36,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:36,129 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1217 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:36,129 [Thread-5398] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162318ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:36,129 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:36,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1078: begin an election at term 1218 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:36,168 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1078: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1218, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:36,184 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1078: begin an election at term 1219 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:36,200 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1078: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1219, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1219] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1219, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:36,200 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1219 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:36,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:36,200 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:36,357 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_2 | 2019-10-15 03:16:32,317 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection694 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:36,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:36,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1219 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:36,357 [Thread-5404] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162546ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:36,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:36,371 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1079: begin an election at term 1220 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:36,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1079: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1220, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1220] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1220, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:36,396 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1220 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:36,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:36,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:36,525 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:128ms, electionTimeout:128ms datanode_1 | 2019-10-15 03:17:36,525 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:36,525 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1220 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:36,525 [Thread-5408] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162714ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:36,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:36,552 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1080: begin an election at term 1221 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:36,580 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1080: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1221, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1221] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1221, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:36,580 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1221 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:36,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:36,580 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:36,634 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:36,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:36,634 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1221 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:36,634 [Thread-5412] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 162823ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:36,634 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,318 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection694: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t772, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:36,675 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1081: begin an election at term 1222 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,318 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:36,738 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1081: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1222, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1222] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1222, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:36,739 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1222 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:36,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:36,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:36,877 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_1 | 2019-10-15 03:17:36,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:36,877 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1222 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:36,878 [Thread-5416] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163066ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:36,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:36,884 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1082: begin an election at term 1223 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:36,901 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1082: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1223, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1223] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1223, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:36,901 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1223 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:36,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:36,901 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:37,025 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_1 | 2019-10-15 03:17:37,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:37,025 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1223 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:37,025 [Thread-5420] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163214ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:37,025 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:37,038 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1083: begin an election at term 1224 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:37,066 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1083: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1224, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1224] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1224, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:37,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1224 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:37,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:37,067 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,318 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:37,174 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:37,174 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:37,174 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1224 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:37,174 [Thread-5424] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163363ms datanode_1 | 2019-10-15 03:17:37,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:37,188 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1084: begin an election at term 1225 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:37,217 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1084: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1225, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1225] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1225, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:37,217 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1225 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:37,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:37,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:37,306 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:37,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:37,306 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1225 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:37,306 [Thread-5428] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163495ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:37,306 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:37,320 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1085: begin an election at term 1226 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:37,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1085: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1226, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1226] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1226, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:37,336 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1226 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:37,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:37,336 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:37,464 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:37,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,318 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 772 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:37,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1226 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:37,464 [Thread-5432] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163653ms datanode_2 | 2019-10-15 03:16:32,318 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:37,465 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,328 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 758 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:37,476 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1086: begin an election at term 1227 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:37,505 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1086: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1227, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1227] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1227, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:37,505 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1227 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,329 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:37,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,364 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_1 | 2019-10-15 03:17:37,505 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,364 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,365 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 772 for changeToCandidate datanode_1 | 2019-10-15 03:17:37,669 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:163ms, electionTimeout:163ms datanode_1 | 2019-10-15 03:17:37,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:37,669 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1227 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,391 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection695: begin an election at term 773 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:37,669 [Thread-5436] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163858ms datanode_2 | 2019-10-15 03:16:32,392 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 759 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:32,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:37,669 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,392 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:37,683 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1087: begin an election at term 1228 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,394 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection695 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:37,710 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1087: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1228, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1228] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1228, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:37,711 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1228 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:37,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:37,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:37,760 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_1 | 2019-10-15 03:17:37,760 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:37,760 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1228 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:37,760 [Thread-5440] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 163949ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:37,761 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:37,786 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1088: begin an election at term 1229 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:37,814 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1088: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1229, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1229] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1229, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:37,814 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1229 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:37,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,395 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection695 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:37,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:37,913 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:37,913 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:37,913 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1229 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:37,914 [Thread-5444] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164102ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:37,914 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:37,939 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1089: begin an election at term 1230 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:37,966 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1089: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1230, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1230] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1230, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:37,966 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1230 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:37,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:37,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:38,119 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:38,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:38,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1230 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:38,120 [Thread-5448] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164308ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:38,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:38,134 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1090: begin an election at term 1231 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:38,162 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1090: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1231, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1231] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1231, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:38,162 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1231 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:38,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:38,162 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:38,241 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:16:32,396 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection695: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t773, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,396 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:38,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:38,242 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1231 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:38,242 [Thread-5452] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164431ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:38,242 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:38,268 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1091: begin an election at term 1232 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,283 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1091: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1232, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,286 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1091: begin an election at term 1233 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:38,293 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1091: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1233, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1233] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1233, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,293 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1233 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:38,293 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:38,293 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:38,435 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_1 | 2019-10-15 03:17:38,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:38,435 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1233 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:38,435 [Thread-5458] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164624ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:38,435 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:38,461 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1092: begin an election at term 1234 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,396 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:38,490 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1092: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1234, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1234] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1234, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:38,490 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1234 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:38,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:38,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:38,605 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:38,605 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:38,606 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1234 for changeToCandidate datanode_1 | 2019-10-15 03:17:38,606 [Thread-5462] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164795ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:38,606 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:38,630 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1093: begin an election at term 1235 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,658 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1093: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1235, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1235] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1235, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,658 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1235 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:38,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:38,658 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:38,783 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:38,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:38,783 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1235 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:38,783 [Thread-5466] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 164972ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:38,784 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:38,798 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1094: begin an election at term 1236 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:38,815 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1094: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1236, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1236] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1236, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:38,815 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1236 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:38,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:38,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:38,832 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:16ms, electionTimeout:16ms datanode_2 | 2019-10-15 03:16:32,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 773 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:38,832 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,523 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_1 | 2019-10-15 03:17:38,832 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1236 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:38,832 [Thread-5470] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165021ms datanode_1 | 2019-10-15 03:17:38,833 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:38,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1095: begin an election at term 1237 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,523 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 773 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:38,874 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1095: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1237, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1237] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1237, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,538 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection696: begin an election at term 774 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:38,874 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1237 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,540 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 760 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:38,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,540 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:38,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,540 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:38,896 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_2 | 2019-10-15 03:16:32,542 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection696 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:38,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:38,896 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1237 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:38,896 [Thread-5474] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165085ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:38,896 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:38,910 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1096: begin an election at term 1238 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:38,938 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1096: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1238, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1238] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1238, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:38,938 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1238 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:38,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:38,938 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:39,065 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:39,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:39,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1238 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:39,065 [Thread-5478] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165254ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:39,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:39,079 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1097: begin an election at term 1239 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:39,100 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1097: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1239, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1239] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1239, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:39,100 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1239 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:39,100 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,542 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection696 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:39,101 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:39,280 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:39,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:39,280 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1239 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:39,281 [Thread-5482] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165469ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:39,281 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:39,295 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1098: begin an election at term 1240 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:39,323 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1098: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1240, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1240] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1240, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:39,323 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1240 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:39,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:39,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:39,515 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:39,515 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:39,515 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1240 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:39,516 [Thread-5486] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165704ms datanode_1 | 2019-10-15 03:17:39,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:39,542 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1099: begin an election at term 1241 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:39,557 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1099: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1241, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1241] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1241, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:39,558 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1241 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:39,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:39,558 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:39,596 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:39,596 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:39,596 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1241 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:39,596 [Thread-5490] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 165785ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:39,597 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection696: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t774, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:39,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1100: begin an election at term 1242 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,543 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:39,629 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1100: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1242, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1242] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1242, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:39,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1242 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:39,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:39,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:39,821 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_1 | 2019-10-15 03:17:39,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:39,821 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1242 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:39,821 [Thread-5494] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166010ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:39,821 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:39,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1101: begin an election at term 1243 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:39,862 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1101: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1243, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1243] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1243, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:39,863 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1243 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:39,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:39,863 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:40,015 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_1 | 2019-10-15 03:17:40,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:40,015 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1243 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:40,016 [Thread-5498] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166204ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:40,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:40,030 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1102: begin an election at term 1244 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:40,059 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1102: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1244, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1244] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1244, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,543 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:40,059 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1244 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:40,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:40,059 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:40,086 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:40,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:40,086 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1244 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:40,086 [Thread-5502] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166275ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:40,087 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:40,111 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1103: begin an election at term 1245 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:40,134 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1103: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1245, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1245] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1245, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:40,134 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1245 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:40,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:40,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:40,280 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:40,280 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:40,281 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1245 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:40,281 [Thread-5506] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166469ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:40,281 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:40,313 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1104: begin an election at term 1246 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:40,322 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1104: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1246, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1246] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1246, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 774 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:40,322 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1246 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:40,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,650 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:17:40,323 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:40,418 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:95ms, electionTimeout:95ms datanode_2 | 2019-10-15 03:16:32,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,650 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 774 for changeToCandidate datanode_1 | 2019-10-15 03:17:40,418 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,650 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:40,418 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1246 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,659 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection697: begin an election at term 775 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:40,418 [Thread-5511] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166607ms datanode_1 | 2019-10-15 03:17:40,419 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:40,453 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1105: begin an election at term 1247 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,661 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 761 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:40,481 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1105: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1247, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1247] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1247, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:40,482 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1247 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:40,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:40,482 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,661 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:40,552 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_2 | 2019-10-15 03:16:32,663 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection697 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:40,552 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:40,552 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1247 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:40,553 [Thread-5515] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166741ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:40,553 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:40,579 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1106: begin an election at term 1248 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:40,607 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1106: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1248, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1248] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1248, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:40,607 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1248 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:40,607 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:40,607 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:40,678 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:70ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:40,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:40,678 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1248 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:40,678 [Thread-5519] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166867ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:40,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:40,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1107: begin an election at term 1249 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:40,715 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1107: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1249, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1249] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1249, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:40,715 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1249 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:40,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:40,715 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,664 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection697 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:40,757 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:40,757 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:40,757 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1249 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:40,758 [Thread-5523] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 166946ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:40,758 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:40,783 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1108: begin an election at term 1250 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:40,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1108: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1250, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1250] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1250, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:40,800 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1250 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:40,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:40,800 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:40,986 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:186ms, electionTimeout:186ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:40,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:40,986 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1250 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:40,987 [Thread-5527] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167175ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:40,987 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,015 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1109: begin an election at term 1251 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:41,045 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1109: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1251, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1251] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1251, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:41,045 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1251 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:41,045 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:41,045 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,664 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection697: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t775, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:41,056 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_1 | 2019-10-15 03:17:41,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,664 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:41,056 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1251 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,057 [Thread-5531] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167245ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:41,057 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:41,082 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1110: begin an election at term 1252 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:41,098 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1110: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1252, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1252] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1252, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:41,098 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1252 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:41,098 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:41,177 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:78ms, electionTimeout:78ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:41,177 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:41,177 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1252 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:41,177 [Thread-5535] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167366ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,178 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:41,192 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1111: begin an election at term 1253 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:41,220 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1111: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1253, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1253] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1253, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:41,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1253 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:41,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:41,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,664 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:41,271 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:41,271 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1253 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:41,271 [Thread-5539] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167460ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:41,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:41,296 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1112: begin an election at term 1254 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:41,300 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1112: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1254, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:41,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1112: begin an election at term 1255 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:41,338 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1112: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1255, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1255] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1255, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:41,339 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1255 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:41,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:41,339 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:41,448 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_1 | 2019-10-15 03:17:41,448 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:41,448 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1255 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:41,449 [Thread-5545] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167637ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:41,463 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1113: begin an election at term 1256 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,478 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1113: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1256, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1256] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1256, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:41,478 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1256 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:41,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:41,479 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,665 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 775 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:41,565 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | 2019-10-15 03:16:32,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:41,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,665 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:41,565 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1256 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,772 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 762 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:41,566 [Thread-5549] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167754ms datanode_2 | 2019-10-15 03:16:32,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:41,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:32,772 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:41,581 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1114: begin an election at term 1257 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,792 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | 2019-10-15 03:17:41,598 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1114: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1257, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1257] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1257, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 775 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:32,806 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection698: begin an election at term 776 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:41,598 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1257 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:41,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,809 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection698 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:41,598 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,737 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:139ms, electionTimeout:139ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:41,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:41,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1257 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:41,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:41,738 [Thread-5553] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 167926ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:41,764 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1115: begin an election at term 1258 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,790 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1115: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1258, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1258] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1258, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:41,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1258 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:41,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:41,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:41,875 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:41,875 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,875 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1258 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:41,876 [Thread-5557] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168064ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:41,876 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:41,890 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1116: begin an election at term 1259 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:41,906 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1116: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1259, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1259] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1259, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:41,906 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1259 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,809 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection698 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:41,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:41,906 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:42,033 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:42,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:42,033 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1259 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:42,033 [Thread-5561] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168222ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:42,033 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:42,069 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1117: begin an election at term 1260 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,086 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1117: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1260, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1260] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1260, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:42,086 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1260 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:42,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:42,086 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:42,117 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:42,117 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:42,117 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1260 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:42,118 [Thread-5565] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168306ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:42,118 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:42,140 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1118: begin an election at term 1261 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:42,155 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1118: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1261, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1261] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1261, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection698: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t776, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,810 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:42,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1261 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:42,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:42,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:42,215 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:42,215 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:42,215 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1261 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:42,216 [Thread-5569] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168404ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:42,216 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:42,229 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1119: begin an election at term 1262 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:42,251 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1119: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1262, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:42,259 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1119: begin an election at term 1263 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:42,287 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1119: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1263, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1263] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1263, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,287 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1263 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:42,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:42,287 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:42,387 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:42,387 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:42,387 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1263 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:42,388 [Thread-5575] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168576ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:42,388 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,810 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:42,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1120: begin an election at term 1264 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:42,429 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1120: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1264, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1264] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1264, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1264 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:42,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:42,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:42,535 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:42,535 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:42,536 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1264 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:42,536 [Thread-5579] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168724ms datanode_1 | 2019-10-15 03:17:42,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:42,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1121: begin an election at term 1265 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,579 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1121: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1265, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1265] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1265, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:42,579 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1265 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:42,579 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:42,579 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:42,695 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_1 | 2019-10-15 03:17:42,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:42,695 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1265 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:42,695 [Thread-5583] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168884ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:42,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:42,721 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1122: begin an election at term 1266 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,738 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1122: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1266, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1266] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1266, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:42,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1266 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,810 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 776 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:42,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:42,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:42,786 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:48ms, electionTimeout:48ms datanode_2 | 2019-10-15 03:16:32,959 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:148ms, electionTimeout:148ms datanode_1 | 2019-10-15 03:17:42,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:32,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:42,787 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1266 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,959 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 776 for changeToCandidate datanode_1 | 2019-10-15 03:17:42,787 [Thread-5588] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 168976ms datanode_2 | 2019-10-15 03:16:32,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:42,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:42,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1123: begin an election at term 1267 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,978 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection699: begin an election at term 777 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:42,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1123: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1267, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1267] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1267, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 763 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:42,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1267 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:42,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:42,836 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:32,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:42,839 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_1 | 2019-10-15 03:17:42,839 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:42,839 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1267 for changeToCandidate datanode_1 | 2019-10-15 03:17:42,839 [Thread-5592] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169028ms datanode_1 | 2019-10-15 03:17:42,840 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:42,853 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1124: begin an election at term 1268 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,982 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection699 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:42,869 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1124: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1268, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1268] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1268, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:42,869 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1268 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:42,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:42,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:42,942 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:72ms, electionTimeout:72ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:42,942 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:42,942 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1268 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:42,942 [Thread-5596] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169131ms datanode_1 | 2019-10-15 03:17:42,942 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:42,956 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1125: begin an election at term 1269 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:42,974 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1125: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1269, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1269] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1269, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:42,974 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1269 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:42,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:42,974 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:43,074 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:43,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:43,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1269 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:43,075 [Thread-5600] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169263ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:43,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:43,114 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1126: begin an election at term 1270 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:43,133 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1126: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1270, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1270] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1270, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:43,133 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1270 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,982 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection699 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:43,133 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,133 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:43,333 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:43,333 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:43,333 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1270 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:43,334 [Thread-5604] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169522ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:43,360 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1127: begin an election at term 1271 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:43,387 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1127: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1271, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1271] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1271, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:43,387 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1271 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:43,387 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:43,387 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:43,487 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:43,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:43,487 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1271 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:43,487 [Thread-5608] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169676ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:43,487 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:43,501 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1128: begin an election at term 1272 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:43,529 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1128: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1272, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1272] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1272, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,983 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection699: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t777, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:43,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1272 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,983 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:43,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:43,645 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:43,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:43,645 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1272 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:43,645 [Thread-5612] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169834ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:43,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:43,651 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1129: begin an election at term 1273 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,659 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1129: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1273, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1273] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1273, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:43,659 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1273 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:43,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:43,659 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:43,673 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:13ms, electionTimeout:13ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:43,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:43,673 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1273 for changeToCandidate datanode_1 | 2019-10-15 03:17:43,673 [Thread-5616] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169862ms datanode_1 | 2019-10-15 03:17:43,673 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:43,678 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1130: begin an election at term 1274 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:43,694 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1130: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1274, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1274] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1274, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:43,694 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1274 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:43,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:32,983 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:43,694 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,754 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:43,754 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:43,754 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1274 for changeToCandidate datanode_1 | 2019-10-15 03:17:43,754 [Thread-5620] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 169943ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:43,755 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:43,768 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1131: begin an election at term 1275 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:43,777 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1131: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1275, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1275] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1275, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:43,777 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1275 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:43,777 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:43,777 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:43,933 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_1 | 2019-10-15 03:17:43,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:43,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1275 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:43,933 [Thread-5624] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170122ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:43,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:43,959 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1132: begin an election at term 1276 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:43,975 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1132: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1276, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1276] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1276, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:43,976 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1276 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:43,976 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:43,976 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:44,154 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:44,154 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:44,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1276 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,983 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 777 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:44,155 [Thread-5628] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170343ms datanode_2 | 2019-10-15 03:16:32,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:44,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:44,168 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1133: begin an election at term 1277 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,983 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:44,208 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1133: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1277, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1277] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1277, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:32,996 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 764 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:44,209 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1277 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:32,996 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:44,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:32,996 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:44,209 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:44,359 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_1 | 2019-10-15 03:17:44,359 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:44,359 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1277 for changeToCandidate datanode_2 | 2019-10-15 03:16:32,996 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:44,360 [Thread-5632] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170548ms datanode_2 | 2019-10-15 03:16:33,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_1 | 2019-10-15 03:17:44,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 777 for changeToCandidate datanode_1 | 2019-10-15 03:17:44,373 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1134: begin an election at term 1278 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:33,148 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection700: begin an election at term 778 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,150 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 765 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:44,394 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1134: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1278, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:44,402 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1134: begin an election at term 1279 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:44,430 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1134: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1279, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1279] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1279, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,150 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:44,430 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1279 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,152 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection700 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:44,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:44,430 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:44,490 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:44,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:44,490 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1279 for changeToCandidate datanode_1 | 2019-10-15 03:17:44,490 [Thread-5638] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170679ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:44,490 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:44,504 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1135: begin an election at term 1280 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:44,520 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1135: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1280, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1280] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1280, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:44,520 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1280 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:44,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:44,520 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:44,530 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:44,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:44,530 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1280 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:44,531 [Thread-5642] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170719ms datanode_1 | 2019-10-15 03:17:44,531 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:44,545 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1136: begin an election at term 1281 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,152 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection700 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:44,572 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1136: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1281, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1281] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1281, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:44,572 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1281 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:44,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:44,573 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:44,745 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:44,745 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:44,745 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1281 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:44,745 [Thread-5646] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 170934ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:44,746 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:44,752 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1137: begin an election at term 1282 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:44,754 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1137: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1282, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:44,759 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1137: begin an election at term 1283 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:44,783 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1137: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1283, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1283] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1283, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:44,783 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1283 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:44,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:44,783 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:44,902 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:44,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:44,902 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1283 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:44,902 [Thread-5652] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171091ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:44,902 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:44,918 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1138: begin an election at term 1284 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,153 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection700: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t778, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:44,934 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1138: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1284, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1284] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1284, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,153 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:44,934 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1284 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:44,934 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:44,934 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:45,014 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:80ms, electionTimeout:80ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:45,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:45,015 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1284 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:45,015 [Thread-5656] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171204ms datanode_1 | 2019-10-15 03:17:45,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:45,061 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1139: begin an election at term 1285 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:45,078 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1139: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1285, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1285] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1285, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:45,078 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1285 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:45,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:45,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:45,138 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:45,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,138 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1285 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,138 [Thread-5660] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171327ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:45,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:45,164 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1140: begin an election at term 1286 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:45,179 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1140: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1286, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1286] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1286, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,153 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:45,179 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1286 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:45,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:45,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:45,189 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:45,189 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:45,189 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1286 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:45,189 [Thread-5664] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171378ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:45,189 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:45,203 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1141: begin an election at term 1287 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:45,230 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1141: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1287, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1287] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1287, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:45,231 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1287 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:45,231 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,234 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:45,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:45,234 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1287 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,153 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 778 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:45,234 [Thread-5668] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171423ms datanode_2 | 2019-10-15 03:16:33,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:45,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:45,248 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1142: begin an election at term 1288 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,183 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 766 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:45,264 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1142: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1288, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1288] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1288, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,183 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:45,264 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1288 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:45,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,183 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:45,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:45,443 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:178ms, electionTimeout:178ms datanode_2 | 2019-10-15 03:16:33,183 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:45,443 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,207 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_1 | 2019-10-15 03:17:45,443 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1288 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,207 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 778 for changeToCandidate datanode_1 | 2019-10-15 03:17:45,443 [Thread-5672] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171632ms datanode_2 | 2019-10-15 03:16:33,207 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:45,443 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,221 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection701: begin an election at term 779 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:45,457 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1143: begin an election at term 1289 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:45,485 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1143: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1289, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1289] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1289, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:45,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1289 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:45,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:45,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,224 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection701 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:45,642 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:156ms, electionTimeout:156ms datanode_1 | 2019-10-15 03:17:45,642 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:45,642 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1289 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:45,642 [Thread-5676] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 171831ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:45,642 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:45,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1144: begin an election at term 1290 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:45,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1144: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1290, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1290] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1290, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:45,674 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1290 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:45,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:45,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:45,847 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:45,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:45,847 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1290 for changeToCandidate datanode_1 | 2019-10-15 03:17:45,847 [Thread-5680] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172036ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:45,847 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,860 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1145: begin an election at term 1291 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:45,888 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1145: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1291, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1291] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1291, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:45,888 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1291 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:45,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:45,888 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:45,981 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | 2019-10-15 03:16:33,227 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection701 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:45,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:45,981 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1291 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:45,981 [Thread-5684] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172170ms datanode_1 | 2019-10-15 03:17:45,981 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:46,007 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1146: begin an election at term 1292 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:46,024 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1146: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1292, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1292] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1292, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:46,024 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1292 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:46,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:46,024 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:46,071 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:46,071 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:46,071 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1292 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:46,071 [Thread-5688] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172260ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:46,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:46,085 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1147: begin an election at term 1293 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:46,103 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1147: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1293, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1293] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1293, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:46,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1293 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:46,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:46,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:33,227 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection701: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t779, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,227 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:46,245 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:46,245 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:46,245 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1293 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:46,246 [Thread-5692] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172434ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:46,246 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:46,271 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1148: begin an election at term 1294 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:46,287 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1148: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1294, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1294] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1294, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:46,288 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1294 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:46,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:46,288 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:46,322 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:46,322 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:46,322 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1294 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:46,322 [Thread-5696] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172511ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:46,322 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:46,336 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1149: begin an election at term 1295 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:46,352 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1149: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1295, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1295] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1295, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:46,352 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1295 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:46,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:46,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:46,476 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:123ms, electionTimeout:123ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:46,476 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:46,476 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1295 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,228 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:46,476 [Thread-5700] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172665ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:46,477 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:46,490 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1150: begin an election at term 1296 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:46,516 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1150: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1296, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:46,521 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1150: begin an election at term 1297 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:46,536 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1150: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1297, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1297] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1297, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:46,536 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1297 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:46,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:46,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:46,614 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:77ms, electionTimeout:77ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:46,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:46,614 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1297 for changeToCandidate datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:46,614 [Thread-5706] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172803ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:46,614 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:46,629 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1151: begin an election at term 1298 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:46,656 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1151: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1298, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1298] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1298, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:46,656 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1298 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:46,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:46,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:46,731 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | 2019-10-15 03:16:33,228 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 779 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:46,731 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:46,731 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1298 for changeToCandidate datanode_1 | 2019-10-15 03:17:46,731 [Thread-5710] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 172920ms datanode_1 | 2019-10-15 03:17:46,732 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,228 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:46,746 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1152: begin an election at term 1299 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,378 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 767 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:46,773 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1152: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1299, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1299] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1299, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:46,774 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1299 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,378 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:46,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,378 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:46,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,419 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:191ms, electionTimeout:191ms datanode_1 | 2019-10-15 03:17:46,877 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:103ms, electionTimeout:103ms datanode_2 | 2019-10-15 03:16:33,419 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:46,877 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,420 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 779 for changeToCandidate datanode_1 | 2019-10-15 03:17:46,877 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1299 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,420 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:46,877 [Thread-5714] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173066ms datanode_2 | 2019-10-15 03:16:33,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection702: begin an election at term 780 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:46,878 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,438 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection702 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:46,890 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1153: begin an election at term 1300 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:46,909 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1153: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1300, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:46,919 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1153: begin an election at term 1301 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:46,945 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1153: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1301, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1301] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1301, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:46,945 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1301 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:46,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:46,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:46,987 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:46,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:46,988 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1301 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:46,988 [Thread-5720] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173177ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:46,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:47,002 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1154: begin an election at term 1302 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:47,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1154: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1302, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1302] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1302, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:47,032 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1302 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:47,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:47,032 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:47,176 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:47,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:47,176 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1302 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:47,176 [Thread-5724] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173365ms datanode_1 | 2019-10-15 03:17:47,176 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection702 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:47,215 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1155: begin an election at term 1303 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:47,244 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1155: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1303, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1303] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1303, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:47,244 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1303 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:47,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:47,244 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:47,368 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:47,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:47,369 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1303 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:47,369 [Thread-5728] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173558ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:47,369 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:47,394 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1156: begin an election at term 1304 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:47,411 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1156: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1304, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1304] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1304, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:47,411 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1304 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:47,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:47,411 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:47,486 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:47,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:47,486 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1304 for changeToCandidate datanode_1 | 2019-10-15 03:17:47,486 [Thread-5732] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173675ms datanode_1 | 2019-10-15 03:17:47,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:47,512 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1157: begin an election at term 1305 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:47,529 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1157: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1305, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1305] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1305, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:47,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1305 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,439 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection702: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t780, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:47,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:47,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,439 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:47,585 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:55ms, electionTimeout:55ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:47,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:47,585 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1305 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:47,585 [Thread-5736] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 173774ms datanode_1 | 2019-10-15 03:17:47,585 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:47,599 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1158: begin an election at term 1306 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:47,615 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1158: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1306, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1306] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1306, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:47,616 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1306 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:47,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:47,616 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:47,811 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:195ms, electionTimeout:195ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:47,811 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:47,811 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1306 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:47,812 [Thread-5740] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174000ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:47,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:47,828 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1159: begin an election at term 1307 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:47,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1159: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1307, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1307] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1307, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:47,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1307 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:47,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:47,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,439 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:47,985 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:47,985 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:47,985 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1307 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:47,985 [Thread-5744] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174174ms datanode_1 | 2019-10-15 03:17:47,986 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:47,999 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1160: begin an election at term 1308 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:48,023 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1160: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1308, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1308] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1308, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:48,023 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1308 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:48,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:48,023 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:48,182 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:158ms, electionTimeout:158ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:48,182 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:48,182 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1308 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:48,182 [Thread-5748] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174371ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:48,182 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:48,208 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1161: begin an election at term 1309 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:48,235 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1161: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1309, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1309] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1309, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:48,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1309 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,440 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 780 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:48,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:48,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,440 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:48,310 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:74ms, electionTimeout:74ms datanode_2 | 2019-10-15 03:16:33,617 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 768 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:48,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,617 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:48,310 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1309 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,617 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:33,617 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:48,310 [Thread-5752] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174499ms datanode_2 | 2019-10-15 03:16:33,634 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_1 | 2019-10-15 03:17:48,310 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:48,324 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1162: begin an election at term 1310 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,634 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:48,352 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1162: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1310, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1310] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1310, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,634 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 780 for changeToCandidate datanode_1 | 2019-10-15 03:17:48,352 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1310 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:48,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,649 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection703: begin an election at term 781 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,352 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,652 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection703 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:48,398 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:48,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:48,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1310 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:48,398 [Thread-5756] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174587ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:48,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:48,413 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1163: begin an election at term 1311 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,431 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1163: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1311, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1311] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1311, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,431 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1311 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:48,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:48,431 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:48,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:48,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:48,447 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1311 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:48,447 [Thread-5760] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174635ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:48,447 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:48,452 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1164: begin an election at term 1312 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:48,460 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1164: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1312, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1312] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1312, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:48,460 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1312 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:48,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:48,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:48,655 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:48,655 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:48,655 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1312 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:48,656 [Thread-5764] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174844ms datanode_2 | 2019-10-15 03:16:33,653 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection703 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:48,656 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:48,670 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1165: begin an election at term 1313 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:48,687 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1165: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1313, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1313] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1313, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:48,687 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1313 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:48,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:48,687 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:48,773 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:85ms, electionTimeout:85ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:48,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:48,773 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1313 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:48,773 [Thread-5768] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 174962ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:48,774 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:48,787 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1166: begin an election at term 1314 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,815 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1166: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1314, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1314] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1314, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,815 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1314 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:48,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:48,815 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:48,842 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_1 | 2019-10-15 03:17:48,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:48,842 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1314 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:48,842 [Thread-5772] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175031ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:48,842 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:48,856 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1167: begin an election at term 1315 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:48,873 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1167: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1315, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1315] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1315, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:48,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1315 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:48,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,653 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection703: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t781, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:48,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,653 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:49,007 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:49,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:49,007 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1315 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:49,007 [Thread-5776] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175196ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:49,007 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:49,032 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1168: begin an election at term 1316 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:49,048 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1168: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1316, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1316] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1316, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:49,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1316 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:49,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:49,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:49,179 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:49,179 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:49,179 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1316 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:49,180 [Thread-5780] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175368ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:49,180 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:49,194 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1169: begin an election at term 1317 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,222 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1169: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1317, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1317] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1317, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:49,222 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1317 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:49,222 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:49,231 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:49,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,653 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:49,232 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1317 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:49,232 [Thread-5784] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175421ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:49,232 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:49,245 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1170: begin an election at term 1318 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:49,262 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1170: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1318, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1318] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1318, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:49,262 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1318 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:49,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:49,262 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:49,312 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:50ms, electionTimeout:50ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:49,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:49,312 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1318 for changeToCandidate datanode_1 | 2019-10-15 03:17:49,312 [Thread-5788] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175501ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:49,313 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:49,337 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1171: begin an election at term 1319 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:49,344 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1171: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1319, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:49,353 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1171: begin an election at term 1320 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,360 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1171: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1320, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1320] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1320, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:49,360 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1320 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:49,360 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:49,436 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_1 | 2019-10-15 03:17:49,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:49,436 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1320 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,654 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 781 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:49,436 [Thread-5794] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175625ms datanode_2 | 2019-10-15 03:16:33,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:49,436 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,654 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:33,675 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_1 | 2019-10-15 03:17:49,460 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1172: begin an election at term 1321 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,675 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:49,496 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1172: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1321, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1321] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1321, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:49,497 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1321 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,675 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 781 for changeToCandidate datanode_1 | 2019-10-15 03:17:49,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:49,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,689 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection704: begin an election at term 782 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:49,599 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_2 | 2019-10-15 03:16:33,693 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection704 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:49,599 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:49,599 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1321 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:49,599 [Thread-5798] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175788ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:49,600 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:49,613 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1173: begin an election at term 1322 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:49,635 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1173: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1322, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1322] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1322, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:49,635 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1322 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:49,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:49,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:49,678 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:49,678 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:49,678 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1322 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:49,678 [Thread-5802] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175867ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:49,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1174: begin an election at term 1323 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:49,707 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1174: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1323, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1323] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1323, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,708 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1323 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:49,708 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,693 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection704 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:49,708 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:49,711 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:3ms, electionTimeout:3ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:49,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:49,711 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1323 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:49,711 [Thread-5806] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 175900ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:49,711 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:49,724 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1175: begin an election at term 1324 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:49,752 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1175: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1324, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1324] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1324, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:49,752 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1324 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:49,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:49,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:49,933 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:49,933 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:49,933 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1324 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,934 [Thread-5810] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176122ms datanode_1 | 2019-10-15 03:17:49,934 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:49,948 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1176: begin an election at term 1325 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:49,976 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1176: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1325, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1325] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1325, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:49,976 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1325 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:49,976 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:49,976 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:50,146 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:170ms, electionTimeout:170ms datanode_2 | 2019-10-15 03:16:33,693 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection704: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t782, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,693 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:50,146 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:50,147 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1325 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:50,147 [Thread-5814] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176336ms datanode_1 | 2019-10-15 03:17:50,147 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:50,173 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1177: begin an election at term 1326 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:50,229 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1177: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1326, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1326] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1326, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:50,229 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1326 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:50,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:50,229 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:50,374 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:145ms, electionTimeout:145ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:50,374 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:50,375 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1326 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:50,375 [Thread-5818] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176563ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:50,375 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:50,401 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1178: begin an election at term 1327 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:50,429 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1178: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1327, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1327] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1327, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:50,429 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1327 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:50,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:50,429 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:50,460 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:50,460 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,693 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:50,461 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1327 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:50,461 [Thread-5822] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176649ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:50,461 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:50,475 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1179: begin an election at term 1328 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:50,491 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1179: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1328, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1328] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1328, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:50,491 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1328 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:50,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:50,491 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:50,610 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_1 | 2019-10-15 03:17:50,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:50,610 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1328 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:50,610 [Thread-5826] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176799ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:50,610 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:50,624 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1180: begin an election at term 1329 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:50,624 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1180: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1329, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:50,650 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1180: begin an election at term 1330 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:50,666 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1180: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1330, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1330] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1330, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:50,666 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1330 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:50,666 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:50,666 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:50,772 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:50,772 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:50,772 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1330 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:50,772 [Thread-5832] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 176961ms datanode_1 | 2019-10-15 03:17:50,772 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:50,778 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1181: begin an election at term 1331 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,693 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 782 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:50,786 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1181: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1331, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1331] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1331, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,784 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:90ms, electionTimeout:90ms datanode_1 | 2019-10-15 03:17:50,786 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1331 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:50,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,784 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 782 for changeToCandidate datanode_1 | 2019-10-15 03:17:50,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,784 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:50,939 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:153ms, electionTimeout:153ms datanode_2 | 2019-10-15 03:16:33,789 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection705: begin an election at term 783 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:50,940 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:50,940 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1331 for changeToCandidate datanode_1 | 2019-10-15 03:17:50,940 [Thread-5836] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177129ms datanode_2 | 2019-10-15 03:16:33,790 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 769 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:50,940 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:33,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:50,954 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1182: begin an election at term 1332 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:50,965 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1182: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1332, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1332] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1332, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,790 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:50,965 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1332 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:50,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:50,965 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,792 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection705 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:50,988 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:50,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:50,988 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1332 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:50,988 [Thread-5840] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177177ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:50,988 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:51,009 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1183: begin an election at term 1333 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:51,038 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1183: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1333, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1333] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1333, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:51,038 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1333 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:51,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:51,038 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:51,049 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:51,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:51,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1333 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:51,049 [Thread-5844] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177238ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:51,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:51,063 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1184: begin an election at term 1334 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,081 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1184: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1334, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1334] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1334, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:51,082 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1334 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,082 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:51,082 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:51,233 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:151ms, electionTimeout:151ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:51,234 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,793 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection705 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:51,234 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1334 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:51,234 [Thread-5848] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177423ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:51,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:51,258 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: begin an election at term 1335 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:51,271 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1335, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:51,293 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: begin an election at term 1336 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:51,309 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1336, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:51,332 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: begin an election at term 1337 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:51,428 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1185: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1337, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1337] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1337, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:51,428 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1337 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:51,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:51,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:51,454 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:26ms, electionTimeout:26ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:51,454 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:51,454 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1337 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:51,455 [Thread-5856] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177643ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:51,474 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1186: begin an election at term 1338 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:51,502 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1186: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1338, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1338] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1338, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:51,502 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1338 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:51,502 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,793 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection705: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t783, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:51,502 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,793 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:51,571 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:68ms, electionTimeout:68ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:51,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:51,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1338 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:51,571 [Thread-5860] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177760ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:51,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:51,585 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1187: begin an election at term 1339 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:51,601 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1187: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1339, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1339] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1339, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:51,602 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1339 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:51,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:51,602 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:51,752 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:150ms, electionTimeout:150ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:51,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:51,752 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1339 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,753 [Thread-5864] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 177941ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,753 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:51,783 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1188: begin an election at term 1340 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:51,811 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1188: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1340, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1340] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1340, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:51,811 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1340 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:51,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,793 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:51,812 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:51,927 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:51,927 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:51,927 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1340 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:51,928 [Thread-5868] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178116ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:51,928 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:51,953 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1189: begin an election at term 1341 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:51,969 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1189: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1341, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1341] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1341, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:51,969 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1341 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:51,969 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:51,969 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:51,984 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:14ms, electionTimeout:14ms datanode_1 | 2019-10-15 03:17:51,984 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,984 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1341 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:51,984 [Thread-5872] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178173ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:51,984 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:51,998 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1190: begin an election at term 1342 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:52,015 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1190: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1342, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1342] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1342, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:52,015 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1342 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,793 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 783 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:52,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:52,015 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,793 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:52,055 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:39ms, electionTimeout:39ms datanode_2 | 2019-10-15 03:16:33,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:52,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,843 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:49ms, electionTimeout:49ms datanode_2 | 2019-10-15 03:16:33,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,843 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 783 for changeToCandidate datanode_1 | 2019-10-15 03:17:52,055 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1342 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:33,856 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection706: begin an election at term 784 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:52,055 [Thread-5876] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178244ms datanode_2 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection706 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:52,055 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:52,079 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1191: begin an election at term 1343 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:52,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1191: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1343, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:52,094 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1191: begin an election at term 1344 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:52,122 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1191: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1344, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1344] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1344, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:52,122 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1344 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:52,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:52,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:52,198 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:75ms, electionTimeout:75ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:52,198 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:52,198 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1344 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:52,198 [Thread-5882] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178387ms datanode_1 | 2019-10-15 03:17:52,198 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:52,224 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1192: begin an election at term 1345 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:52,233 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1192: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1345, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1345] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1345, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:52,233 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1345 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:52,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:52,233 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:52,363 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:130ms, electionTimeout:130ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:52,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:52,363 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1345 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:52,364 [Thread-5886] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178552ms datanode_2 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection706 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:52,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:52,378 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1193: begin an election at term 1346 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:52,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1193: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1346, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1346] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1346, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:52,396 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1346 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:52,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:52,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:52,485 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:52,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:52,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1346 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:52,486 [Thread-5890] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178674ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:52,486 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:52,500 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1194: begin an election at term 1347 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:52,515 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1194: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1347, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1347] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1347, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:52,516 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1347 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:52,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:52,516 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:52,684 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:52,684 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:52,684 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1347 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection706: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t784, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:52,685 [Thread-5894] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 178873ms datanode_2 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:52,685 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:52,712 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1195: begin an election at term 1348 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:52,740 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1195: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1348, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1348] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1348, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:52,741 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1348 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:52,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:52,741 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:52,873 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:132ms, electionTimeout:132ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:52,873 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:52,873 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1348 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:52,874 [Thread-5898] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179062ms datanode_1 | 2019-10-15 03:17:52,874 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:52,888 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1196: begin an election at term 1349 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:52,904 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1196: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1349, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1349] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1349, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:52,904 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1349 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:52,904 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:52,905 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:53,101 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:53,101 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:53,102 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1349 for changeToCandidate datanode_1 | 2019-10-15 03:17:53,102 [Thread-5902] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179291ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:53,125 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1197: begin an election at term 1350 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,142 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1197: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1350, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1350] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1350, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:53,142 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1350 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:53,142 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:53,142 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,860 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:53,235 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:92ms, electionTimeout:92ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:53,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:53,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1350 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:53,235 [Thread-5906] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179424ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:53,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:53,249 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1198: begin an election at term 1351 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:53,266 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1198: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1351, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1351] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1351, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:53,266 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1351 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:53,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:53,266 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:53,311 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:53,311 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:53,311 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1351 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:53,312 [Thread-5910] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179500ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:53,312 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:53,326 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1199: begin an election at term 1352 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:53,342 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1199: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1352, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1352] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1352, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1352 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:53,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:53,406 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:53,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:53,406 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1352 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,860 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 784 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:53,406 [Thread-5914] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179595ms datanode_1 | 2019-10-15 03:17:53,406 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:53,420 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1200: begin an election at term 1353 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,860 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,860 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:33,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 770 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:33,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:53,449 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1200: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1353, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1353] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1353, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:53,449 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1353 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,861 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:53,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:33,902 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_1 | 2019-10-15 03:17:53,449 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:33,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,902 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 784 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,902 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:53,577 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:127ms, electionTimeout:127ms datanode_1 | 2019-10-15 03:17:53,577 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:33,916 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection707: begin an election at term 785 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:53,577 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1353 for changeToCandidate datanode_2 | 2019-10-15 03:16:33,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection707 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:53,577 [Thread-5918] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179766ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:53,577 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:53,592 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1201: begin an election at term 1354 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:53,598 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1201: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1354, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:53,607 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1201: begin an election at term 1355 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:53,635 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1201: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1355, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1355] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1355, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:53,635 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1355 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:53,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:53,635 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:53,738 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:53,738 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:53,738 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1355 for changeToCandidate datanode_1 | 2019-10-15 03:17:53,738 [Thread-5924] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 179927ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:53,739 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:53,754 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1202: begin an election at term 1356 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:53,782 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1202: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1356, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1356] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1356, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,782 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1356 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:53,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:53,782 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:53,911 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:53,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:53,912 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1356 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:53,912 [Thread-5928] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 180101ms datanode_2 | 2019-10-15 03:16:33,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection707 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:53,912 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:53,926 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1203: begin an election at term 1357 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:53,941 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1203: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1357, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1357] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1357, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:53,941 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1357 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:53,941 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:53,941 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,040 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:54,040 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:54,040 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1357 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:54,040 [Thread-5932] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 180229ms datanode_1 | 2019-10-15 03:17:54,040 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:54,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1204: begin an election at term 1358 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:54,072 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1204: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1358, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1358] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1358, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:54,072 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1358 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:54,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,072 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:54,182 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: addNew group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-BD8D864B3B3A:java.util.concurrent.CompletableFuture@269fb90d[Not completed] datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,185 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: new RaftServerImpl for group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:33,919 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection707: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t785, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_2 | 2019-10-15 03:16:33,919 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:54,186 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:54,186 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:54,187 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:54,187 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a does not exist. Creating ... datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:54,201 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/in_use.lock acquired by nodename 7@2783c458fa57 datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:54,214 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a has been successfully formatted. datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,214 [pool-8-thread-1] INFO - group-BD8D864B3B3A: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:54,215 INFO segmented.SegmentedRaftLogWorker: new 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | 2019-10-15 03:16:33,920 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_1 | 2019-10-15 03:17:54,216 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:54,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:54,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:54,221 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-BD8D864B3B3A,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:54,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:54,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:54,270 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1358 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:54,270 [Thread-5936] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 180459ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:54,279 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: changes role from FOLLOWER to FOLLOWER at term 1 for recognizeCandidate:cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:54,279 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:54,279 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:54,295 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1205: begin an election at term 1359 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,920 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 785 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:54,314 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1205: Election TIMEOUT; received 1 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1359] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1359, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:54,327 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1205: begin an election at term 1360 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:54,336 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: change Leader from null to cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at term 1 for appendEntries, leader elected after 121ms datanode_2 | 2019-10-15 03:16:33,958 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 771 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:33,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:54,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1205: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1360, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1360] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1360, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:33,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:54,348 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1360 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:33,958 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:54,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,032 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:112ms, electionTimeout:112ms datanode_1 | 2019-10-15 03:17:54,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:54,354 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 datanode_2 | 2019-10-15 03:16:34,032 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:54,354 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | 2019-10-15 03:16:34,032 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 785 for changeToCandidate datanode_1 | 2019-10-15 03:17:54,385 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-BD8D864B3B3A-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/current/log_inprogress_0 datanode_2 | 2019-10-15 03:16:34,033 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:54,446 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_2 | 2019-10-15 03:16:34,058 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection708: begin an election at term 786 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:54,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:54,446 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1360 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,061 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection708 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:54,446 [Thread-5945] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 180635ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,446 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:54,458 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1206: begin an election at term 1361 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:54,465 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1206: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1361, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:54,492 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1206: begin an election at term 1362 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:54,532 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1206: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1362, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1362] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1362, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,532 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1362 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:54,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:54,532 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:54,695 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:162ms, electionTimeout:162ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:54,695 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:54,695 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1362 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,695 [Thread-5951] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 180884ms datanode_1 | 2019-10-15 03:17:54,696 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:54,711 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1207: begin an election at term 1363 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:54,736 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1207: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1363, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1363] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1363, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:54,737 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1363 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:54,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:54,737 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:54,894 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:54,894 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:54,894 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1363 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,061 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection708 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:54,894 [Thread-5955] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181083ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:54,895 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:54,908 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1208: begin an election at term 1364 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:54,937 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1208: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1364, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1364] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1364, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:54,937 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1364 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:54,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:54,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:55,093 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:155ms, electionTimeout:155ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:55,093 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:55,093 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1364 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:55,094 [Thread-5959] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181282ms datanode_1 | 2019-10-15 03:17:55,094 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:55,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1209: begin an election at term 1365 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:55,135 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1209: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1365, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1365] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1365, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:55,135 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1365 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:55,135 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:55,135 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:55,319 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_2 | 2019-10-15 03:16:34,062 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection708: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t786, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:55,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,062 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:55,319 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1365 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:55,319 [Thread-5963] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181508ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:55,319 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:55,333 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1210: begin an election at term 1366 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:55,349 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1210: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1366, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1366] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1366, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:55,350 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1366 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:55,350 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:55,350 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:55,355 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:55,355 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:55,355 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1366 for changeToCandidate datanode_1 | 2019-10-15 03:17:55,355 [Thread-5967] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181544ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:55,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:55,379 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1211: begin an election at term 1367 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:55,396 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1211: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1367, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1367] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1367, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:55,396 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1367 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:55,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:55,396 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:55,501 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:104ms, electionTimeout:104ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:55,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:55,501 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1367 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,062 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:55,501 [Thread-5971] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181690ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:55,501 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:55,531 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1212: begin an election at term 1368 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:55,548 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1212: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1368, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1368] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1368, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:55,548 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1368 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:55,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:55,548 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:55,637 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:55,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:55,637 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1368 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:55,637 [Thread-5975] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181826ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:55,637 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:55,651 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1213: begin an election at term 1369 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:55,670 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1213: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1369, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1369] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1369, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:55,670 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1369 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:55,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:55,670 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:55,722 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:55,722 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:55,723 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1369 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:55,723 [Thread-5980] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181911ms datanode_2 | 2019-10-15 03:16:34,062 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 786 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:55,723 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:55,726 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1214: begin an election at term 1370 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:55,735 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1214: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1370, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1370] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1370, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,153 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 772 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:55,735 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1370 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:55,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,153 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:55,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,153 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:55,791 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:56ms, electionTimeout:56ms datanode_2 | 2019-10-15 03:16:34,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:131ms, electionTimeout:131ms datanode_1 | 2019-10-15 03:17:55,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:55,791 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1370 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,194 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 786 for changeToCandidate datanode_1 | 2019-10-15 03:17:55,792 [Thread-5985] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 181980ms datanode_1 | 2019-10-15 03:17:55,797 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:55,842 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1215: begin an election at term 1371 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,218 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection709: begin an election at term 787 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:55,857 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1215: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1371, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1371] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1371, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,219 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 773 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:55,857 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1371 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:55,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:55,857 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,219 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:56,030 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:172ms, electionTimeout:172ms datanode_2 | 2019-10-15 03:16:34,221 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection709 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:56,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:56,030 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1371 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:56,030 [Thread-5989] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182219ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:56,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:56,057 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1216: begin an election at term 1372 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:56,078 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1216: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1372, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1372] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1372, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:56,078 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1372 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:56,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:56,078 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:56,090 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:12ms, electionTimeout:12ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:56,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:56,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1372 for changeToCandidate datanode_1 | 2019-10-15 03:17:56,091 [Thread-5993] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182279ms datanode_1 | 2019-10-15 03:17:56,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:56,106 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1217: begin an election at term 1373 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:56,122 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1217: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1373, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1373] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1373, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:56,122 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1373 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:56,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:56,122 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:56,264 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:56,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:56,264 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1373 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:56,264 [Thread-5997] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182453ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:56,264 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:56,278 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1218: begin an election at term 1374 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:56,295 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1218: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1374, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1374] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1374, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,222 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection709 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:56,295 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1374 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:56,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:56,295 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:56,398 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:102ms, electionTimeout:102ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:56,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:56,398 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1374 for changeToCandidate datanode_1 | 2019-10-15 03:17:56,398 [Thread-6001] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182587ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:56,398 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:56,423 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1219: begin an election at term 1375 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:56,451 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1219: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1375, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1375] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1375, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:56,451 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1375 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:56,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:56,451 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:56,594 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:143ms, electionTimeout:143ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:56,594 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:56,594 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1375 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:56,594 [Thread-6005] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182783ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:56,595 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:56,620 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1220: begin an election at term 1376 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:56,648 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1220: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1376, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1376] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1376, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,222 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection709: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t787, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:56,648 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1376 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,222 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:56,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:56,648 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:56,802 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_1 | 2019-10-15 03:17:56,802 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:56,802 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1376 for changeToCandidate datanode_1 | 2019-10-15 03:17:56,802 [Thread-6009] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 182991ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:56,803 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:56,943 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1221: begin an election at term 1377 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:56,970 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1221: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1377, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1377] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1377, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:56,971 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1377 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:56,971 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:56,971 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:57,060 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:89ms, electionTimeout:89ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:57,060 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:57,060 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1377 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:57,061 [Thread-6013] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183249ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:57,061 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:57,065 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1222: begin an election at term 1378 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,073 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1222: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1378, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1378] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1378, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:57,073 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1378 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,073 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:57,073 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:57,083 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:57,083 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:57,083 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1378 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:57,083 [Thread-6018] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183272ms datanode_2 | 2019-10-15 03:16:34,222 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:57,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,090 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1223: begin an election at term 1379 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:57,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1223: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1379, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1379] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1379, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:57,099 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1379 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:57,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:57,099 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:57,134 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:0ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:57,134 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1379 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:57,136 [Thread-6022] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183323ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:57,138 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:57,146 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1224: begin an election at term 1380 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:57,155 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1224: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1380, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1380] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1380, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:57,155 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1380 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:57,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,155 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:57,330 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:57,330 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1380 for changeToCandidate datanode_1 | 2019-10-15 03:17:57,330 [Thread-6026] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183519ms datanode_1 | 2019-10-15 03:17:57,330 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:57,339 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1225: begin an election at term 1381 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:57,355 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1225: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1381, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1381] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1381, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,222 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 787 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:57,355 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1381 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,222 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:57,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,222 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:57,356 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,358 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 774 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:34,358 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:57,377 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:21ms, electionTimeout:21ms datanode_2 | 2019-10-15 03:16:34,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:34,359 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:57,377 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,417 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_1 | 2019-10-15 03:17:57,377 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1381 for changeToCandidate datanode_1 | 2019-10-15 03:17:57,378 [Thread-6030] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183566ms datanode_2 | 2019-10-15 03:16:34,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:57,378 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,417 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 787 for changeToCandidate datanode_1 | 2019-10-15 03:17:57,383 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1226: begin an election at term 1382 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:57,394 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1226: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1382, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1382] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1382, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,417 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:17:57,395 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1382 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,431 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection710: begin an election at term 788 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:57,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,434 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection710 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:57,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,594 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:57,594 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:57,594 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1382 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:57,594 [Thread-6034] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183783ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:57,595 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,620 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1227: begin an election at term 1383 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:57,636 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1227: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1383, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1383] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1383, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:57,636 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1383 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:57,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:57,636 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:57,786 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:57,786 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:57,786 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1383 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,786 [Thread-6038] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 183975ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:57,787 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:57,800 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1228: begin an election at term 1384 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:57,816 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1228: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1384, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1384] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1384, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:34,436 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection710 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,816 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1384 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:57,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:57,816 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:57,931 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_1 | 2019-10-15 03:17:57,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:57,931 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1384 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:57,931 [Thread-6042] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184120ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:57,931 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:57,944 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1229: begin an election at term 1385 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:57,961 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1229: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1385, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1385] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1385, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:57,961 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1385 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:57,961 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:57,961 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:58,035 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:58,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:58,035 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1385 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,437 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection710: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t788, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:58,035 [Thread-6046] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184224ms datanode_2 | 2019-10-15 03:16:34,437 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:58,035 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:58,049 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1230: begin an election at term 1386 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:58,064 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1230: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1386, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1386] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1386, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:58,065 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1386 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:58,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,065 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:58,217 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:58,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:58,218 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1386 for changeToCandidate datanode_1 | 2019-10-15 03:17:58,218 [Thread-6050] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184406ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:58,218 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:58,237 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1231: begin an election at term 1387 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:58,254 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1231: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1387, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1387] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1387, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:58,254 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1387 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:58,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,254 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:58,327 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:73ms, electionTimeout:73ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:58,327 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:58,328 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1387 for changeToCandidate datanode_1 | 2019-10-15 03:17:58,328 [Thread-6054] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184517ms datanode_1 | 2019-10-15 03:17:58,328 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:58,341 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1232: begin an election at term 1388 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:58,357 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1232: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1388, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1388] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1388, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:58,357 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1388 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:58,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,357 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:58,472 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:114ms, electionTimeout:114ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:58,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,437 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:58,472 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1388 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:58,472 [Thread-6058] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184661ms datanode_1 | 2019-10-15 03:17:58,472 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:58,486 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1233: begin an election at term 1389 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:58,504 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1233: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1389, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1389] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1389, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:58,504 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1389 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:58,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:58,504 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:58,620 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:58,620 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:58,620 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1389 for changeToCandidate datanode_1 | 2019-10-15 03:17:58,620 [Thread-6062] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184809ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:58,620 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:58,634 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1234: begin an election at term 1390 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:58,650 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1234: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1390, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1390] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1390, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:58,650 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1390 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:58,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,650 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:58,679 [Thread-6066] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184868ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:58,679 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:28ms, electionTimeout:28ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:58,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:58,679 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1390 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:58,679 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:58,692 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1235: begin an election at term 1391 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:58,707 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1235: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1391, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1391] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1391, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:58,707 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1391 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:58,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,707 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:58,790 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_1 | 2019-10-15 03:17:58,790 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:58,790 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1391 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:58,790 [Thread-6070] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 184979ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:58,791 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:34,437 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 788 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,804 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1236: begin an election at term 1392 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,437 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:58,820 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1236: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1392, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1392] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1392, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,467 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 775 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:34,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:58,820 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1392 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:58,820 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:58,820 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,467 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:17:58,850 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:29ms, electionTimeout:29ms datanode_1 | 2019-10-15 03:17:58,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:58,850 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1392 for changeToCandidate datanode_1 | 2019-10-15 03:17:58,850 [Thread-6074] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185039ms datanode_1 | 2019-10-15 03:17:58,850 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:58,864 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1237: begin an election at term 1393 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,524 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:86ms, electionTimeout:86ms datanode_2 | 2019-10-15 03:16:34,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,524 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 788 for changeToCandidate datanode_1 | 2019-10-15 03:17:58,890 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1237: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1393, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1393] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1393, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:34,537 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 776 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:17:58,890 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1393 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:17:58,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,537 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:17:58,890 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,537 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:34,548 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection711: begin an election at term 789 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,016 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | 2019-10-15 03:16:34,552 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection711 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:59,016 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,016 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1393 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:59,017 [Thread-6078] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185205ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:59,017 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:59,031 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1238: begin an election at term 1394 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:59,047 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1238: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1394, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1394] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1394, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:59,047 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1394 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:59,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,047 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:59,186 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:138ms, electionTimeout:138ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:59,186 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:59,186 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1394 for changeToCandidate datanode_1 | 2019-10-15 03:17:59,186 [Thread-6082] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185375ms datanode_1 | 2019-10-15 03:17:59,186 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:59,200 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1239: begin an election at term 1395 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:59,217 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1239: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1395, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1395] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1395, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:59,217 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1395 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:59,217 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:59,218 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:59,362 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:144ms, electionTimeout:144ms datanode_1 | 2019-10-15 03:17:59,362 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,362 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1395 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,362 [Thread-6086] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185551ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:59,363 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:59,368 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1240: begin an election at term 1396 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,374 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1240: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1396, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1396] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1396, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,374 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1396 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:59,374 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:59,374 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:59,380 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:5ms, electionTimeout:5ms datanode_1 | 2019-10-15 03:17:59,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:59,380 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1396 for changeToCandidate datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:59,380 [Thread-6090] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185569ms datanode_2 | 2019-10-15 03:16:34,553 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection711 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:17:59,380 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:59,385 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1241: begin an election at term 1397 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,395 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1241: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1397, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1397] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1397, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,395 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1397 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:59,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,395 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:59,529 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:59,529 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:59,529 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1397 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:59,529 [Thread-6094] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185718ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,530 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:59,544 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1242: begin an election at term 1398 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:59,561 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1242: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1398, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1398] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1398, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:59,561 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1398 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:59,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:17:59,561 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:17:59,595 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:34ms, electionTimeout:34ms datanode_1 | 2019-10-15 03:17:59,595 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:17:59,595 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1398 for changeToCandidate datanode_1 | 2019-10-15 03:17:59,595 [Thread-6098] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185784ms datanode_1 | 2019-10-15 03:17:59,596 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,609 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1243: begin an election at term 1399 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,624 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1243: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1399, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1399] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1399, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:59,624 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1399 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:59,624 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:59,624 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,553 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection711: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t789, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:17:59,625 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:0ms, electionTimeout:0ms datanode_2 | 2019-10-15 03:16:34,553 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:17:59,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,625 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1399 for changeToCandidate datanode_1 | 2019-10-15 03:17:59,625 [Thread-6102] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 185814ms datanode_1 | 2019-10-15 03:17:59,625 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:17:59,639 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1244: begin an election at term 1400 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:17:59,654 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1244: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1400, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1400] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1400, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:17:59,654 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1400 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:17:59,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:59,654 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:17:59,822 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:168ms, electionTimeout:168ms datanode_1 | 2019-10-15 03:17:59,822 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:17:59,822 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1400 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:17:59,822 [Thread-6106] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186011ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:17:59,823 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:17:59,836 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1245: begin an election at term 1401 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,859 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1245: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1401, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:17:59,875 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1245: begin an election at term 1402 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:17:59,892 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1245: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1402, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1402] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1402, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:17:59,892 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1402 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:17:59,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:17:59,892 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,920 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:27ms, electionTimeout:27ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:17:59,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:17:59,920 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1402 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:17:59,920 [Thread-6112] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186109ms datanode_1 | 2019-10-15 03:17:59,920 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:17:59,934 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1246: begin an election at term 1403 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:17:59,949 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1246: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1403, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1403] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1403, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,554 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:17:59,949 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1403 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:17:59,949 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:17:59,949 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:00,103 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:154ms, electionTimeout:154ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:00,103 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:00,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1403 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:00,104 [Thread-6116] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186292ms datanode_1 | 2019-10-15 03:18:00,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:00,134 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1247: begin an election at term 1404 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:00,150 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1247: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1404, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1404] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1404, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:00,150 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1404 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:00,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:00,150 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:00,270 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:00,270 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:00,271 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1404 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:00,271 [Thread-6120] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186459ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:00,271 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:00,285 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1248: begin an election at term 1405 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:00,301 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1248: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1405, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1405] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1405, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:00,301 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1405 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:00,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:00,301 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:00,422 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:120ms, electionTimeout:120ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:00,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:00,422 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1405 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,554 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 789 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:00,422 [Thread-6124] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186611ms datanode_2 | 2019-10-15 03:16:34,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:00,422 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:00,434 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1249: begin an election at term 1406 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,554 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:00,450 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1249: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1406, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1406] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1406, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,584 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 777 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:18:00,450 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1406 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:00,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:00,450 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,584 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:18:00,458 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:16:34,661 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:106ms, electionTimeout:106ms datanode_1 | 2019-10-15 03:18:00,458 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:00,458 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1406 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:00,459 [Thread-6128] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186647ms datanode_2 | 2019-10-15 03:16:34,661 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 789 for changeToCandidate datanode_1 | 2019-10-15 03:18:00,459 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,661 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:18:00,472 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1250: begin an election at term 1407 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,675 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection712: begin an election at term 790 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:00,488 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1250: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1407, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1407] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1407, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,678 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection712 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:00,488 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1407 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:00,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:00,488 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:18:00,496 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:00,496 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:00,496 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1407 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:00,497 [Thread-6132] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186685ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:00,497 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:00,511 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1251: begin an election at term 1408 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:00,526 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1251: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1408, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1408] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1408, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:00,526 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1408 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:00,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:00,526 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:00,536 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:10ms, electionTimeout:10ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:00,536 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:00,536 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1408 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:00,537 [Thread-6136] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186725ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:00,537 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:00,550 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1252: begin an election at term 1409 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,678 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection712 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:00,564 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1252: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1409, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1409] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1409, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:00,564 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1409 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:00,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:00,564 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:00,729 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:164ms, electionTimeout:164ms datanode_1 | 2019-10-15 03:18:00,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:00,729 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1409 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:00,729 [Thread-6140] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 186918ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:00,729 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:00,761 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1253: begin an election at term 1410 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:00,777 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1253: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1410, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1410] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1410, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:00,777 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1410 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:00,777 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:00,777 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:00,924 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:146ms, electionTimeout:146ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:00,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:00,924 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1410 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:00,924 [Thread-6144] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187113ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:00,924 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,679 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection712: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t790, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:00,938 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1254: begin an election at term 1411 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,679 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:00,954 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1254: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1411, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1411] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1411, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:00,954 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1411 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:00,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:00,954 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:01,073 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:119ms, electionTimeout:119ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:01,073 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:01,073 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1411 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:01,074 [Thread-6148] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187262ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:01,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:01,099 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1255: begin an election at term 1412 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:01,114 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1255: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1412, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1412] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1412, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:01,115 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1412 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:01,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:01,115 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:01,166 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:01,166 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:01,166 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1412 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:01,166 [Thread-6152] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187355ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:01,167 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:01,180 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1256: begin an election at term 1413 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,679 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:18:01,195 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1256: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1413, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1413] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1413, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:01,195 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1413 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:01,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:01,195 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:01,259 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:01,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:01,259 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1413 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:01,259 [Thread-6156] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187448ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:01,259 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:01,273 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1257: begin an election at term 1414 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:01,280 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1257: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1414, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:01,312 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1257: begin an election at term 1415 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:01,337 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1257: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1415, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1415] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1415, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:01,337 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1415 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:01,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:01,337 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:01,455 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:01,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:01,455 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1415 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:01,455 [Thread-6162] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187644ms datanode_1 | 2019-10-15 03:18:01,455 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:01,469 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1258: begin an election at term 1416 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,679 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 790 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:01,485 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1258: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1416, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1416] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1416, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,679 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:01,485 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1416 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,679 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:01,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:34,717 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 778 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:34,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:01,485 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,717 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:01,565 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:16:34,717 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:18:01,565 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,738 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 779 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:18:01,565 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1416 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:01,565 [Thread-6166] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187754ms datanode_2 | 2019-10-15 03:16:34,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:01,566 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,738 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:18:01,586 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1259: begin an election at term 1417 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:100ms, electionTimeout:100ms datanode_1 | 2019-10-15 03:18:01,593 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1259: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1417, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:01,602 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1259: begin an election at term 1418 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 790 for changeToCandidate datanode_1 | 2019-10-15 03:18:01,618 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1259: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1418, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1418] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1418, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:18:01,618 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1418 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,794 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection713: begin an election at term 791 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,798 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection713 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:01,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:01,618 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:01,776 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:157ms, electionTimeout:157ms datanode_1 | 2019-10-15 03:18:01,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:01,776 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1418 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:01,776 [Thread-6172] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 187965ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:01,776 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:01,790 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1260: begin an election at term 1419 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:01,805 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1260: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1419, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1419] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1419, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:01,805 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1419 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:01,805 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:01,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:01,869 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:01,869 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,798 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection713 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:01,869 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1419 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:01,869 [Thread-6176] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188058ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:01,870 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:01,895 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1261: begin an election at term 1420 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:01,911 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1261: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1420, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1420] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1420, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:01,911 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1420 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:01,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:01,911 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:02,030 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:118ms, electionTimeout:118ms datanode_1 | 2019-10-15 03:18:02,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:02,030 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1420 for changeToCandidate datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:02,030 [Thread-6180] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188219ms datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:02,030 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:02,052 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1262: begin an election at term 1421 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:02,068 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1262: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1421, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1421] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1421, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:02,068 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1421 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:02,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:02,068 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:18:02,080 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:11ms, electionTimeout:11ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:02,080 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:02,080 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1421 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,080 [Thread-6184] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188269ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:02,080 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,094 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1263: begin an election at term 1422 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:02,109 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1263: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1422, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1422] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1422, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:02,109 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1422 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:02,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:02,109 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:02,204 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:94ms, electionTimeout:94ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:02,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,798 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection713: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t791, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,798 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:02,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1422 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:02,204 [Thread-6188] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188393ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:02,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:02,218 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1264: begin an election at term 1423 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:02,235 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1264: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1423, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1423] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1423, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:02,235 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1423 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:02,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:02,235 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:02,333 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_1 | 2019-10-15 03:18:02,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:02,334 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1423 for changeToCandidate datanode_1 | 2019-10-15 03:18:02,334 [Thread-6192] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188523ms datanode_1 | 2019-10-15 03:18:02,334 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:02,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1265: begin an election at term 1424 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:02,363 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1265: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1424, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1424] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1424, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:02,364 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1424 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:02,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:02,364 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:02,427 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:63ms, electionTimeout:63ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:02,427 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:02,427 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1424 for changeToCandidate datanode_1 | 2019-10-15 03:18:02,427 [Thread-6196] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188616ms datanode_1 | 2019-10-15 03:18:02,428 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:02,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1266: begin an election at term 1425 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,452 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1266: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1425, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1425] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1425, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:02,452 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1425 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:02,452 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:02,627 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:02,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,798 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:18:02,627 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1425 for changeToCandidate datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:02,627 [Thread-6200] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188816ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:02,627 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:02,641 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1267: begin an election at term 1426 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:02,657 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1267: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1426, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1426] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1426, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:02,657 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1426 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:02,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:02,657 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:02,806 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:149ms, electionTimeout:149ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:02,806 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:02,806 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1426 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:02,807 [Thread-6204] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 188995ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:02,807 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,820 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1268: begin an election at term 1427 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:02,835 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1268: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1427, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1427] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1427, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:02,835 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1427 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:02,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:02,835 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,799 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 791 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:03,028 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_1 | 2019-10-15 03:18:03,028 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:34,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:03,029 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1427 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:03,029 [Thread-6208] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189218ms datanode_2 | 2019-10-15 03:16:34,916 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 780 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:34,916 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:03,029 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:34,916 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:03,051 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1269: begin an election at term 1428 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,066 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1269: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1428, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1428] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1428, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,066 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1428 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,916 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:18:03,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:03,066 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:34,992 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:193ms, electionTimeout:193ms datanode_1 | 2019-10-15 03:18:03,126 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:60ms, electionTimeout:60ms datanode_1 | 2019-10-15 03:18:03,126 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:03,126 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1428 for changeToCandidate datanode_1 | 2019-10-15 03:18:03,127 [Thread-6212] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189315ms datanode_1 | 2019-10-15 03:18:03,127 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:03,141 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1270: begin an election at term 1429 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,156 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1270: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1429, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1429] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1429, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:34,992 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:03,156 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1429 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:34,992 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 791 for changeToCandidate datanode_2 | 2019-10-15 03:16:34,992 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:18:03,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,006 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection714: begin an election at term 792 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,156 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:35,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection714 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:03,210 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:53ms, electionTimeout:53ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:03,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:03,210 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1429 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:03,210 [Thread-6216] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189399ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:03,210 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:03,224 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1271: begin an election at term 1430 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:03,231 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1271: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1430, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:03,239 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1271: begin an election at term 1431 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:03,255 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1271: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1431, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1431] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1431, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:03,255 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1431 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:03,255 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:03,255 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:03,368 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:113ms, electionTimeout:113ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:03,368 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:03,368 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1431 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:03,369 [Thread-6222] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189557ms datanode_1 | 2019-10-15 03:18:03,369 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:03,391 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1272: begin an election at term 1432 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:03,407 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1272: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1432, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1432] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1432, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:03,407 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1432 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:03,407 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:03,407 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:35,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection714 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:03,603 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:196ms, electionTimeout:196ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:03,603 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:03,603 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1432 for changeToCandidate datanode_1 | 2019-10-15 03:18:03,604 [Thread-6226] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189792ms datanode_1 | 2019-10-15 03:18:03,604 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:03,618 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1273: begin an election at term 1433 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:03,629 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1273: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1433, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1433] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1433, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:03,629 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1433 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:03,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:03,629 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:03,644 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:03,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:03,644 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1433 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:03,645 [Thread-6230] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189833ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:03,645 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:03,658 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1274: begin an election at term 1434 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,674 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1274: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1434, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1434] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1434, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:03,674 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1434 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:03,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:03,674 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:03,772 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:03,772 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection714: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t792, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,772 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1434 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,010 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:18:03,773 [Thread-6234] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 189961ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:03,773 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:03,786 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1275: begin an election at term 1435 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:03,817 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1275: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1435, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:03,830 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1275: begin an election at term 1436 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,846 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1275: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1436, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1436] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1436, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:03,846 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1436 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:03,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:03,846 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:04,019 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:04,019 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:04,020 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1436 for changeToCandidate datanode_1 | 2019-10-15 03:18:04,020 [Thread-6240] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190209ms datanode_1 | 2019-10-15 03:18:04,020 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:04,033 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1276: begin an election at term 1437 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:04,049 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1276: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1437, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1437] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1437, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:04,049 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1437 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:04,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:04,049 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:04,134 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:84ms, electionTimeout:84ms datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:04,134 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:04,134 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1437 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:04,134 [Thread-6244] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190323ms datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:04,135 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:04,148 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1277: begin an election at term 1438 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:04,164 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1277: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1438, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:04,168 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1277: begin an election at term 1439 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:04,204 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1277: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1439, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1439] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1439, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,010 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:18:04,204 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1439 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:04,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:04,204 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:18:04,281 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:76ms, electionTimeout:76ms datanode_1 | 2019-10-15 03:18:04,281 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:04,281 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1439 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:04,281 [Thread-6250] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190470ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:04,281 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:04,331 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1278: begin an election at term 1440 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:04,348 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1278: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1440, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1440] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1440, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:04,348 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1440 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:04,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:04,348 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:04,464 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:04,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:04,464 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1440 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:04,464 [Thread-6254] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190653ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:04,464 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:04,477 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1279: begin an election at term 1441 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:04,484 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1279: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1441, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1441] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1441, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,010 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 792 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:04,484 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1441 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:04,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:04,484 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:35,144 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | 2019-10-15 03:16:35,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:04,621 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_1 | 2019-10-15 03:18:04,621 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,144 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 792 for changeToCandidate datanode_1 | 2019-10-15 03:18:04,621 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1441 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:35,158 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715: begin an election at term 793 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,159 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 781 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:18:04,621 [Thread-6258] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190810ms datanode_2 | 2019-10-15 03:16:35,159 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:04,621 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:35,159 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:04,635 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1280: begin an election at term 1442 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,159 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_1 | 2019-10-15 03:18:04,644 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1280: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1442, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1442] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1442, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,160 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t793, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:04,644 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1442 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,184 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715: begin an election at term 794 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:04,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,186 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:04,644 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:04,752 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:04,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:04,752 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1442 for changeToCandidate datanode_1 | 2019-10-15 03:18:04,752 [Thread-6262] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 190941ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:04,752 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:04,765 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1281: begin an election at term 1443 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:04,781 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1281: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1443, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1443] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1443, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:04,781 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1443 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:04,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:04,781 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:04,955 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:173ms, electionTimeout:173ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:04,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:04,955 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1443 for changeToCandidate datanode_1 | 2019-10-15 03:18:04,955 [Thread-6266] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191144ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:04,955 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:04,981 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1282: begin an election at term 1444 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:04,996 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1282: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1444, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1444] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1444, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:04,996 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1444 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:04,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:04,996 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:05,084 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:88ms, electionTimeout:88ms datanode_1 | 2019-10-15 03:18:05,084 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,186 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:05,084 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1444 for changeToCandidate datanode_1 | 2019-10-15 03:18:05,085 [Thread-6270] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191273ms datanode_1 | 2019-10-15 03:18:05,085 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:05,109 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1283: begin an election at term 1445 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:05,136 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1283: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1445, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1445] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1445, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:05,136 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1445 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:05,136 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,137 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:05,175 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:38ms, electionTimeout:38ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:05,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:05,175 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1445 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:05,175 [Thread-6274] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191364ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,175 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:05,189 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1284: begin an election at term 1446 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:05,201 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1284: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1446, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:05,204 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1284: begin an election at term 1447 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:05,219 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1284: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1447, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1447] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1447, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:05,220 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1447 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:05,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:05,220 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:05,342 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:122ms, electionTimeout:122ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:05,342 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:05,342 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1447 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,187 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection715: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t794, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,187 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:18:05,342 [Thread-6280] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191531ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:05,343 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:05,368 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1285: begin an election at term 1448 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:05,384 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1285: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1448, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1448] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1448, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:05,384 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1448 for DISCOVERED_A_NEW_TERM datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,384 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:05,384 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:05,410 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:25ms, electionTimeout:25ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:05,410 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:05,410 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1448 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:05,410 [Thread-6284] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191599ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:05,410 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:05,425 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1286: begin an election at term 1449 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:05,441 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1286: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1449, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1449] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1449, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:05,442 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1449 for DISCOVERED_A_NEW_TERM datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:05,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,187 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:18:05,442 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,541 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:05,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:05,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1449 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:05,542 [Thread-6288] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191730ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:05,542 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,554 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1287: begin an election at term 1450 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:05,571 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1287: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1450, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1450] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1450, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:05,571 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1450 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:05,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:05,571 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:05,631 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:59ms, electionTimeout:59ms datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:05,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:05,631 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1450 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:05,631 [Thread-6292] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191820ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:05,631 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:05,646 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: begin an election at term 1451 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:05,654 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1451, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:05,661 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: begin an election at term 1452 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:05,666 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1452, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:05,676 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: begin an election at term 1453 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,187 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 794 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:05,691 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1288: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1453, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1453] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1453, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:05,691 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1453 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,187 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,232 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_2 | 2019-10-15 03:16:35,232 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:05,691 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:05,691 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:18:05,735 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_2 | 2019-10-15 03:16:35,233 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 794 for changeToCandidate datanode_1 | 2019-10-15 03:18:05,735 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:05,735 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1453 for changeToCandidate datanode_1 | 2019-10-15 03:18:05,736 [Thread-6300] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 191924ms datanode_2 | 2019-10-15 03:16:35,233 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:18:05,736 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:35,246 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection716: begin an election at term 795 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,249 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection716 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:05,749 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1289: begin an election at term 1454 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,765 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1289: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1454, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1454] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1454, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:05,765 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1454 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:05,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:05,765 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:05,945 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:179ms, electionTimeout:179ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:05,945 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:05,945 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1454 for changeToCandidate datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:05,945 [Thread-6304] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192134ms datanode_1 | 2019-10-15 03:18:05,946 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:05,959 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1290: begin an election at term 1455 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:05,975 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1290: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1455, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1455] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1455, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:05,975 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1455 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:05,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:05,975 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:06,027 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:51ms, electionTimeout:51ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,027 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1455 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:06,027 [Thread-6308] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192216ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:06,027 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:06,041 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1291: begin an election at term 1456 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,250 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection716 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:06,056 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1291: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1456, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1456] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1456, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,056 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1456 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:06,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:06,056 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:06,090 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:33ms, electionTimeout:33ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:06,090 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,090 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1456 for changeToCandidate datanode_1 | 2019-10-15 03:18:06,091 [Thread-6312] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192279ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:06,091 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:06,104 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1292: begin an election at term 1457 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:06,119 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1292: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1457, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1457] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1457, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:06,119 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1457 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:06,119 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,120 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:06,172 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:06,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:06,172 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1457 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,250 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection716: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t795, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:06,172 [Thread-6316] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192361ms datanode_2 | 2019-10-15 03:16:35,250 INFO impl.LeaderElection: Exception 0: {} datanode_1 | 2019-10-15 03:18:06,172 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,186 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1293: begin an election at term 1458 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:06,202 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1293: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1458, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1458] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1458, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:06,202 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1458 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:06,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:06,324 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:06,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:06,324 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1458 for changeToCandidate datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:06,324 [Thread-6320] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192513ms datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:06,324 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:06,349 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1294: begin an election at term 1459 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,371 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1294: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1459, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1459] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1459, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:06,371 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1459 for DISCOVERED_A_NEW_TERM datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:06,371 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:06,541 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:06,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,250 INFO impl.LeaderElection: Exception 1: {} datanode_1 | 2019-10-15 03:18:06,541 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1459 for changeToCandidate datanode_1 | 2019-10-15 03:18:06,541 [Thread-6324] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192730ms datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,541 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_1 | 2019-10-15 03:18:06,566 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1295: begin an election at term 1460 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:06,582 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1295: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1460, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1460] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1460, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:06,582 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1460 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:06,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:06,582 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:06,682 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_1 | 2019-10-15 03:18:06,682 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_1 | 2019-10-15 03:18:06,682 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1460 for changeToCandidate datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:06,682 [Thread-6328] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 192871ms datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:06,682 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:06,703 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1296: begin an election at term 1461 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:06,719 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1296: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1461, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1461] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1461, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:06,719 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1461 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:06,719 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:06,719 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:06,854 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:06,854 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1461 for changeToCandidate datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:06,854 [Thread-6332] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 193043ms datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:06,854 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:06,878 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1297: begin an election at term 1462 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | ... 1 more datanode_1 | 2019-10-15 03:18:06,891 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1297: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1462, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1462] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1462, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 795 for DISCOVERED_A_NEW_TERM datanode_1 | 2019-10-15 03:18:06,891 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1462 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:06,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_1 | 2019-10-15 03:18:06,891 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | 2019-10-15 03:16:35,347 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:96ms, electionTimeout:96ms datanode_1 | 2019-10-15 03:18:06,937 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:45ms, electionTimeout:45ms datanode_1 | 2019-10-15 03:18:06,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_1 | 2019-10-15 03:18:06,937 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1462 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,347 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 795 for changeToCandidate datanode_1 | 2019-10-15 03:18:06,937 [Thread-6336] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 193126ms datanode_2 | 2019-10-15 03:16:35,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_1 | 2019-10-15 03:18:06,937 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | 2019-10-15 03:16:35,361 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection717: begin an election at term 796 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:06,951 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1298: begin an election at term 1463 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,364 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection717 got exception when requesting votes: {} datanode_1 | 2019-10-15 03:18:06,967 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1298: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1463, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1463] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1463, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_1 | 2019-10-15 03:18:06,967 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1463 for DISCOVERED_A_NEW_TERM datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_1 | 2019-10-15 03:18:06,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_1 | 2019-10-15 03:18:06,967 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_1 | 2019-10-15 03:18:07,074 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState: change to CANDIDATE, lastRpcTime:107ms, electionTimeout:107ms datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:18:07,074 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:18:07,074 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from FOLLOWER to CANDIDATE at term 1463 for changeToCandidate datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_1 | 2019-10-15 03:18:07,075 [Thread-6340] ERROR - pipeline Action CLOSE on pipeline PipelineID=1a183fe3-faf2-4917-9518-d5f35bc43a13.Reason : 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c is in candidate state for 193263ms datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_1 | 2019-10-15 03:18:07,075 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start LeaderElection datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_1 | 2019-10-15 03:18:07,088 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1299: begin an election at term 1464 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_1 | 2019-10-15 03:18:07,104 INFO impl.LeaderElection: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-LeaderElection1299: Election REJECTED; received 2 response(s) [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-749b19c7-0772-44d2-8efd-0664e6aa0748#0:FAIL-t1464, 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c<-cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3#0:FAIL-t1464] and 0 exception(s); 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1464, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_1 | 2019-10-15 03:18:07,104 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: changes role from CANDIDATE to FOLLOWER at term 1464 for DISCOVERED_A_NEW_TERM datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_1 | 2019-10-15 03:18:07,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown LeaderElection datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_1 | 2019-10-15 03:18:07,104 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: start FollowerState datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_1 | 2019-10-15 03:18:07,201 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove FOLLOWER 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13:t1464, leader=null, voted=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c, raftlog=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_1 | 2019-10-15 03:18:07,201 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: shutdown datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_1 | 2019-10-15 03:18:07,202 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_1 | 2019-10-15 03:18:07,202 INFO impl.RoleInfo: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: shutdown FollowerState datanode_1 | 2019-10-15 03:18:07,202 INFO impl.FollowerState: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:07,202 INFO impl.StateMachineUpdater: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-StateMachineUpdater: set stopIndex = -1 datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_1 | 2019-10-15 03:18:07,203 INFO impl.RaftServerImpl: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13: closes. applyIndex: -1 datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_1 | 2019-10-15 03:18:07,203 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_1 | 2019-10-15 03:18:07,203 INFO segmented.SegmentedRaftLogWorker: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c@group-D5F35BC43A13-SegmentedRaftLogWorker close() datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_1 | 2019-10-15 03:18:37,204 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove group-D5F35BC43A13:null datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,365 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection717 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_1 | 2019-10-15 03:19:01,683 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove group-D5F35BC43A13:null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_1 | 2019-10-15 03:19:03,066 INFO impl.RaftServerProxy: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: remove group-D5F35BC43A13:null datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,365 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection717: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t796, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,365 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,365 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,365 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 796 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,365 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 782 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:35,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:35,473 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 783 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:35,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,473 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:35,499 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:133ms, electionTimeout:133ms datanode_2 | 2019-10-15 03:16:35,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 796 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:35,554 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection718: begin an election at term 797 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,557 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection718 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,558 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection718 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,558 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection718: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t797, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,558 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,558 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,558 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 797 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,558 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,707 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 784 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:35,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,707 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:35,724 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:165ms, electionTimeout:165ms datanode_2 | 2019-10-15 03:16:35,724 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,724 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 797 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,724 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:35,746 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection719: begin an election at term 798 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,747 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 785 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:35,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,747 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:35,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection719 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,749 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection719 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,750 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection719: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t798, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,750 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,750 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,750 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 798 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,750 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,750 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,866 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_2 | 2019-10-15 03:16:35,867 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,867 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 798 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,867 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:35,893 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection720: begin an election at term 799 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,896 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection720 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection720 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,899 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection720: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t799, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,899 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,899 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,900 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 799 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,901 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | 2019-10-15 03:16:35,901 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,901 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 799 for changeToCandidate datanode_2 | 2019-10-15 03:16:35,901 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:35,914 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection721: begin an election at term 800 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,917 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection721 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,917 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection721 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,918 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection721: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t800, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:35,918 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,918 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:35,918 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 800 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:35,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:35,918 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,984 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 786 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:35,984 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:35,984 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:35,984 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:16:36,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 800 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,076 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection722: begin an election at term 801 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,079 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection722 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,079 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection722 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,079 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection722: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t801, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,079 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,079 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 801 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,080 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,174 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 787 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,174 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,174 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,261 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:181ms, electionTimeout:181ms datanode_2 | 2019-10-15 03:16:36,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,261 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 801 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,278 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection723: begin an election at term 802 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,281 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection723 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,281 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection723 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,281 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection723: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t802, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,281 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,281 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,281 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 802 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,281 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,346 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 788 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,347 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,465 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:183ms, electionTimeout:183ms datanode_2 | 2019-10-15 03:16:36,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 802 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,479 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection724: begin an election at term 803 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,482 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection724 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,482 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection724 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,482 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection724: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t803, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,482 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,482 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,482 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 803 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,483 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 789 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,563 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,597 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 790 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,597 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,681 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 791 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,681 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,681 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:199ms, electionTimeout:199ms datanode_2 | 2019-10-15 03:16:36,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 803 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,706 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection725: begin an election at term 804 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,711 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection725 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,712 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection725 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,712 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection725: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t804, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,712 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,712 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 804 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,803 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 792 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:36,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,804 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:36,839 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:126ms, electionTimeout:126ms datanode_2 | 2019-10-15 03:16:36,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,839 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 804 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,839 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,853 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection726: begin an election at term 805 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,856 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection726 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,856 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection726 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,857 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection726: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t805, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,857 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,857 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,857 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 805 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,857 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,857 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:36,967 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:109ms, electionTimeout:109ms datanode_2 | 2019-10-15 03:16:36,967 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:36,967 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 805 for changeToCandidate datanode_2 | 2019-10-15 03:16:36,967 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:36,980 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727: begin an election at term 806 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,981 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t806, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,994 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727: begin an election at term 807 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,996 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,997 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,997 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection727: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t807, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:36,997 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,997 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:36,997 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 807 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:36,997 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:36,997 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,025 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 793 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,025 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,173 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:16:37,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 807 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,187 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection728: begin an election at term 808 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,190 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection728 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,191 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection728 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,191 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection728: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t808, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,191 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,191 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 808 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,238 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 794 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,238 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,238 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,262 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:70ms, electionTimeout:58ms datanode_2 | 2019-10-15 03:16:37,238 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,262 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 808 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection729: begin an election at term 809 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,292 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 795 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,292 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,292 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,293 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection729 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,297 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection729 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,297 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection729: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t809, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,297 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,298 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 809 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,434 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:136ms, electionTimeout:136ms datanode_2 | 2019-10-15 03:16:37,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,434 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 809 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,434 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,448 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730: begin an election at term 810 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,449 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t810, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,462 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730: begin an election at term 811 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,464 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,465 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,465 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection730: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t811, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,465 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,465 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 811 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,465 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,465 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 796 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,466 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,466 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,524 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:58ms, electionTimeout:58ms datanode_2 | 2019-10-15 03:16:37,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,524 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 811 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,528 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 797 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,528 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,540 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection731: begin an election at term 812 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection731 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,543 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection731 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,544 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection731: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t812, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,544 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,544 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,544 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 812 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 798 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,619 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,668 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:124ms, electionTimeout:124ms datanode_2 | 2019-10-15 03:16:37,668 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,669 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 812 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,682 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection732: begin an election at term 813 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,685 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection732 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,686 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection732 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,686 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection732: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t813, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,686 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,686 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,686 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 813 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,686 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,686 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,729 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 799 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,729 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,730 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,730 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,786 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:99ms, electionTimeout:99ms datanode_2 | 2019-10-15 03:16:37,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,786 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 813 for changeToCandidate datanode_2 | 2019-10-15 03:16:37,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:37,812 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection733: begin an election at term 814 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 800 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,814 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,816 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection733 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,816 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection733 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,817 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection733: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t814, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:37,817 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,817 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:37,817 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 814 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:37,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:37,817 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,900 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 801 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,900 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,900 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:37,935 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 802 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:37,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:37,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:37,935 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,002 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:185ms, electionTimeout:185ms datanode_2 | 2019-10-15 03:16:38,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,003 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 814 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,016 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection734: begin an election at term 815 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,020 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection734 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,020 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection734 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,020 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection734: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t815, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,020 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,020 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,020 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 815 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,020 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,020 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:2ms, electionTimeout:2ms datanode_2 | 2019-10-15 03:16:38,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,023 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 815 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,048 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection735: begin an election at term 816 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 803 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,049 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,049 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,051 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection735 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,051 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection735 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,051 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection735: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t816, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,051 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,051 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,052 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 816 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,094 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | 2019-10-15 03:16:38,094 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,094 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 816 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,094 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,118 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection736: begin an election at term 817 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,121 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection736 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,121 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection736 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,121 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection736: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t817, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,121 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,121 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 817 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,155 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 804 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,155 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,155 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,279 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 805 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,280 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,296 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:174ms, electionTimeout:174ms datanode_2 | 2019-10-15 03:16:38,296 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,296 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 817 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,296 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,308 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection737: begin an election at term 818 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,311 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection737 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection737 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection737: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t818, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,312 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,312 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,312 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 818 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,312 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,395 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:82ms, electionTimeout:82ms datanode_2 | 2019-10-15 03:16:38,395 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,395 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 818 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,395 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,410 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection738: begin an election at term 819 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,413 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection738 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,413 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection738 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,413 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection738: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t819, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,413 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,414 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,414 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 819 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,414 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,507 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 806 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,507 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,507 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,507 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,566 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:152ms, electionTimeout:152ms datanode_2 | 2019-10-15 03:16:38,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,567 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 819 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,580 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection739: begin an election at term 820 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,583 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection739 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,584 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection739 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,584 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection739: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t820, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,584 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,584 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,584 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 820 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,584 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,666 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 807 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,666 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,666 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,666 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:97ms, electionTimeout:97ms datanode_2 | 2019-10-15 03:16:38,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,682 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 820 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,697 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection740: begin an election at term 821 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,699 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection740 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,700 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection740 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,700 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection740: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t821, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,700 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,700 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,700 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 821 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,700 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,709 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:8ms, electionTimeout:8ms datanode_2 | 2019-10-15 03:16:38,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,709 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 821 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,722 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection741: begin an election at term 822 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,725 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection741 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,725 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection741 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,725 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection741: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t822, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,725 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,726 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,726 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 822 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,726 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,726 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,811 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:85ms, electionTimeout:85ms datanode_2 | 2019-10-15 03:16:38,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 822 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,812 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:38,825 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection742: begin an election at term 823 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection742 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection742 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection742: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t823, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:38,828 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,829 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:38,829 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 823 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:38,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:38,829 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,843 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 808 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:38,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,843 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:38,843 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:38,969 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:140ms, electionTimeout:140ms datanode_2 | 2019-10-15 03:16:38,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:38,969 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 823 for changeToCandidate datanode_2 | 2019-10-15 03:16:38,969 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:39,003 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection743: begin an election at term 824 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,004 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 809 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:39,004 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,004 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,005 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:39,006 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection743 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,006 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection743 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,007 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection743: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t824, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,007 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,007 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,007 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 824 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:39,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:39,007 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,078 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:71ms, electionTimeout:71ms datanode_2 | 2019-10-15 03:16:39,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 824 for changeToCandidate datanode_2 | 2019-10-15 03:16:39,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:39,103 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection744: begin an election at term 825 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,106 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection744 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,107 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection744 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,107 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection744: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t825, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,107 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,108 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 825 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:39,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:39,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,173 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 810 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:39,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,173 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,173 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:39,195 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | 2019-10-15 03:16:39,195 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,196 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 825 for changeToCandidate datanode_2 | 2019-10-15 03:16:39,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:39,200 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 811 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:39,200 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,200 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,200 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:39,213 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection745: begin an election at term 826 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,216 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection745 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection745 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,217 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection745: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t826, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:39,217 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,217 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:39,217 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 826 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:39,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:39,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:39,319 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:101ms, electionTimeout:101ms datanode_2 | 2019-10-15 03:16:39,319 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:39,319 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 826 for changeToCandidate datanode_2 | 2019-10-15 03:16:39,319 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,024 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection746: begin an election at term 827 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,025 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 812 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,025 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,026 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,027 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection746 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,028 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection746 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,028 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection746: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t827, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,028 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,028 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,028 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 827 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,028 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,028 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,046 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:17ms, electionTimeout:17ms datanode_2 | 2019-10-15 03:16:40,046 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,046 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 827 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,046 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,053 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection747: begin an election at term 828 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,056 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection747 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,056 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection747 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,056 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection747: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t828, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,056 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,057 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,057 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 828 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,073 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 813 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,073 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,073 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,073 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,080 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | 2019-10-15 03:16:40,080 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,080 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 828 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,080 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,085 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection748: begin an election at term 829 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,086 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 814 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,086 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,086 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,086 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection748 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection748 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,089 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection748: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t829, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,089 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,089 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,089 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 829 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,089 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,136 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | 2019-10-15 03:16:40,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,137 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 829 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,137 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,150 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection749: begin an election at term 830 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,153 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection749 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,154 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection749 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,154 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection749: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t830, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,154 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,154 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,154 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 830 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,154 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,154 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,322 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 815 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,322 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,322 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,330 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:175ms, electionTimeout:175ms datanode_2 | 2019-10-15 03:16:40,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,330 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 830 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,342 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection750: begin an election at term 831 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,345 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection750 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,345 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection750 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,346 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection750: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t831, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,346 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,346 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,346 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 831 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,346 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,346 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,411 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_2 | 2019-10-15 03:16:40,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,411 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 831 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,411 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,425 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection751: begin an election at term 832 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,428 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection751 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,428 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection751 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,429 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection751: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t832, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,429 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,429 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,429 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 832 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,429 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,429 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,496 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 816 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,496 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,517 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:87ms, electionTimeout:87ms datanode_2 | 2019-10-15 03:16:40,517 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,517 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 832 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,517 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,531 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection752: begin an election at term 833 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,534 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection752 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,534 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection752 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,535 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection752: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t833, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,535 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,535 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,535 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 833 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,535 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,535 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 817 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,669 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:134ms, electionTimeout:134ms datanode_2 | 2019-10-15 03:16:40,669 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 833 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,684 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection753: begin an election at term 834 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection753 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection753 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,688 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection753: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t834, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,688 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,689 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,689 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 834 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,689 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,689 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,810 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:121ms, electionTimeout:121ms datanode_2 | 2019-10-15 03:16:40,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,810 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 834 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,824 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection754: begin an election at term 835 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,827 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection754 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection754 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,828 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection754: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t835, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,828 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,828 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,828 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 835 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,828 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 818 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:40,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:40,861 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:40,970 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:141ms, electionTimeout:141ms datanode_2 | 2019-10-15 03:16:40,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:40,970 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 835 for changeToCandidate datanode_2 | 2019-10-15 03:16:40,970 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:40,995 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection755: begin an election at term 836 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,998 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection755 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,998 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection755 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,999 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection755: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t836, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:40,999 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,999 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:40,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 836 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:40,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:40,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,023 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 819 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,023 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,023 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,035 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:35ms, electionTimeout:35ms datanode_2 | 2019-10-15 03:16:41,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,035 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 836 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,059 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection756: begin an election at term 837 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,062 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection756 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,062 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection756 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,062 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection756: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t837, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,062 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,063 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,063 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 837 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,063 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,064 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:1ms, electionTimeout:1ms datanode_2 | 2019-10-15 03:16:41,064 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,064 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 837 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,069 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection757: begin an election at term 838 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,071 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection757 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,071 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection757 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,072 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection757: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t838, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,072 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,072 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,072 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 838 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,104 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 820 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 821 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,121 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,180 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:108ms, electionTimeout:108ms datanode_2 | 2019-10-15 03:16:41,180 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,180 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 838 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,206 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection758: begin an election at term 839 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,208 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection758 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,209 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection758 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,209 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection758: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t839, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,209 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,209 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,210 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 839 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,228 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 822 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,289 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:79ms, electionTimeout:79ms datanode_2 | 2019-10-15 03:16:41,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 839 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,302 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection759: begin an election at term 840 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,305 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection759 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,306 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection759 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,306 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection759: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t840, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,306 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,306 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,306 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 840 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,349 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:42ms, electionTimeout:42ms datanode_2 | 2019-10-15 03:16:41,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 840 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,362 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection760: begin an election at term 841 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,365 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection760 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,365 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection760 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,366 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection760: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t841, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,366 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,366 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,366 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 841 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,366 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,366 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,431 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:64ms, electionTimeout:64ms datanode_2 | 2019-10-15 03:16:41,431 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,431 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 841 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,431 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,457 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection761: begin an election at term 842 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,458 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 823 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,459 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,459 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection761 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,460 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection761 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,460 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection761: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t842, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,460 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,460 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,460 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 842 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,590 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:129ms, electionTimeout:129ms datanode_2 | 2019-10-15 03:16:41,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,590 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 842 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,590 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,603 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection762: begin an election at term 843 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,606 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection762 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,606 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection762 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,607 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection762: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t843, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,607 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,607 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,607 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 843 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,607 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,667 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 824 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,667 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,805 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | 2019-10-15 03:16:41,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,805 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 843 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,818 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection763: begin an election at term 844 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,821 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection763 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,822 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection763 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,822 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection763: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t844, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,822 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,822 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,822 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 844 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,823 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,864 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:41ms, electionTimeout:41ms datanode_2 | 2019-10-15 03:16:41,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,864 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 844 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:41,878 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection764: begin an election at term 845 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,881 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection764 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,882 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection764 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,882 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection764: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t845, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:41,882 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,882 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:41,882 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 845 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:41,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:41,882 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,894 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 825 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:41,894 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,894 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:41,894 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:41,981 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | 2019-10-15 03:16:41,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:41,981 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 845 for changeToCandidate datanode_2 | 2019-10-15 03:16:41,981 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,007 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection765: begin an election at term 846 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection765 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection765 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,010 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection765: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t846, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,010 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,011 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,011 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 846 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,012 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 826 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,012 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,041 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 827 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,041 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,129 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_2 | 2019-10-15 03:16:42,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,129 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 846 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,142 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection766: begin an election at term 847 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,145 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection766 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,145 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection766 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,146 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection766: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t847, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,146 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,146 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,146 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 847 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,146 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,146 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:61ms, electionTimeout:61ms datanode_2 | 2019-10-15 03:16:42,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 847 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,220 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection767: begin an election at term 848 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,222 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection767 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,223 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection767 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,223 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection767: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t848, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,223 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,223 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,223 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 848 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,223 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,260 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:36ms, electionTimeout:36ms datanode_2 | 2019-10-15 03:16:42,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,261 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 848 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,261 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,288 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection768: begin an election at term 849 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,291 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection768 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,309 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection768 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,310 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection768: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t849, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,310 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,310 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,310 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 849 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,310 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 828 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,315 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,453 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:142ms, electionTimeout:142ms datanode_2 | 2019-10-15 03:16:42,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,453 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 849 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,467 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769: begin an election at term 850 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,470 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t850, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,479 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769: begin an election at term 851 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection769: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t851, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,482 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,482 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,482 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 851 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,508 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 829 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,508 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,682 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:200ms, electionTimeout:200ms datanode_2 | 2019-10-15 03:16:42,682 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,683 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 851 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,683 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,698 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection770: begin an election at term 852 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,699 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 830 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,699 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,699 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,699 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,700 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection770 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,701 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection770 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,701 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection770: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t852, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,701 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,701 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 852 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,720 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_2 | 2019-10-15 03:16:42,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,720 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 852 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,724 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection771: begin an election at term 853 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,725 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 831 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,725 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,726 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection771 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,726 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection771 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,727 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection771: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t853, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,727 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,727 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,727 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 853 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,727 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,727 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,794 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:66ms, electionTimeout:66ms datanode_2 | 2019-10-15 03:16:42,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,794 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 853 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,808 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection772: begin an election at term 854 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection772 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection772 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,811 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection772: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t854, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,811 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,811 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 854 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,878 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 832 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:42,878 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,879 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:42,927 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:115ms, electionTimeout:115ms datanode_2 | 2019-10-15 03:16:42,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 854 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:42,942 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection773: begin an election at term 855 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,945 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection773 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,946 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection773 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,946 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection773: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t855, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:42,946 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,946 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:42,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 855 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:42,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:42,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:42,999 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:52ms, electionTimeout:52ms datanode_2 | 2019-10-15 03:16:42,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:42,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 855 for changeToCandidate datanode_2 | 2019-10-15 03:16:42,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,055 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection774: begin an election at term 856 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,057 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection774 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,058 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection774 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,058 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection774: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t856, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,058 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,058 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 856 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,123 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 833 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,123 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,123 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,123 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,225 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:166ms, electionTimeout:166ms datanode_2 | 2019-10-15 03:16:43,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,225 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 856 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection775: begin an election at term 857 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection775 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection775 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection775: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t857, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,242 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,243 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,243 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 857 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,243 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,275 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:32ms, electionTimeout:32ms datanode_2 | 2019-10-15 03:16:43,275 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,276 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 857 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,276 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,310 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection776: begin an election at term 858 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,311 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 834 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,311 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,311 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,312 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection776 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,313 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection776 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,313 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection776: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t858, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,313 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,313 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,313 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 858 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,313 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,430 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:116ms, electionTimeout:116ms datanode_2 | 2019-10-15 03:16:43,430 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,430 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 858 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,430 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,443 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection777: begin an election at term 859 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection777 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection777 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,446 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection777: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t859, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,446 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,447 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,447 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 859 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,447 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,493 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:46ms, electionTimeout:46ms datanode_2 | 2019-10-15 03:16:43,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,493 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 859 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,493 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,507 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection778: begin an election at term 860 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,509 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection778 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,509 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection778 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,510 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection778: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t860, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,510 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,510 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,510 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 860 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,510 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,537 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 835 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,538 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,538 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,538 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,555 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:44ms, electionTimeout:44ms datanode_2 | 2019-10-15 03:16:43,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 860 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,570 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection779: begin an election at term 861 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,572 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection779 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,573 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection779 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,573 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection779: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t861, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,573 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,573 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,573 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 861 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,573 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,645 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 836 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,645 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,645 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,645 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 837 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,672 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,722 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 838 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,722 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,722 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,771 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:197ms, electionTimeout:197ms datanode_2 | 2019-10-15 03:16:43,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,771 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 861 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,776 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection780: begin an election at term 862 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,779 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection780 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,779 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection780 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,779 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection780: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t862, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,780 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,780 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 862 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,803 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:22ms, electionTimeout:22ms datanode_2 | 2019-10-15 03:16:43,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,803 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 862 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,808 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection781: begin an election at term 863 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection781 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,810 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection781 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,811 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection781: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t863, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,811 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,811 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,811 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 863 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,811 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,917 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:105ms, electionTimeout:105ms datanode_2 | 2019-10-15 03:16:43,917 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,917 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 863 for changeToCandidate datanode_2 | 2019-10-15 03:16:43,917 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:43,925 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 839 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,925 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:43,926 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection782: begin an election at term 864 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,928 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection782 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,928 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection782 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,929 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection782: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t864, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:43,929 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,929 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:43,929 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 864 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:43,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:43,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,940 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 840 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:43,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:43,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:43,940 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:192ms, electionTimeout:192ms datanode_2 | 2019-10-15 03:16:44,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,122 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 864 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,136 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 841 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,136 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,137 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection783: begin an election at term 865 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,139 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection783 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection783 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,140 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection783: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t865, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,140 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,140 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,140 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 865 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,140 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,140 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,172 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:31ms, electionTimeout:31ms datanode_2 | 2019-10-15 03:16:44,172 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,172 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 865 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,172 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,186 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection784: begin an election at term 866 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,189 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection784 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,189 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection784 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,189 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection784: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t866, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,189 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,189 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,190 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 866 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:18ms, electionTimeout:18ms datanode_2 | 2019-10-15 03:16:44,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,209 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 866 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,223 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785: begin an election at term 867 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,224 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t867, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,239 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785: begin an election at term 868 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,241 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,242 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection785: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t868, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,242 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,242 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,242 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 868 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,242 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,242 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,257 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:15ms, electionTimeout:15ms datanode_2 | 2019-10-15 03:16:44,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,257 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 868 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection786: begin an election at term 869 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,263 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection786 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,263 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection786 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,263 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection786: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t869, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,263 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,263 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 869 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 842 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,339 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,403 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 843 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,403 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,458 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_2 | 2019-10-15 03:16:44,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,458 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 869 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,458 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,480 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787: begin an election at term 870 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,481 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t870, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,494 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787: begin an election at term 871 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,497 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection787: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t871, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,497 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,497 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,498 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 871 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,527 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 844 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,528 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,528 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,567 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:69ms, electionTimeout:69ms datanode_2 | 2019-10-15 03:16:44,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,567 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 871 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,571 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection788: begin an election at term 872 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,574 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection788 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,575 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection788 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,575 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection788: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t872, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,575 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,575 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,575 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 872 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,575 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,575 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,585 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:9ms, electionTimeout:9ms datanode_2 | 2019-10-15 03:16:44,585 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,585 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 872 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,585 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,589 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection789: begin an election at term 873 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,592 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection789 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,592 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection789 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,592 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection789: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t873, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,592 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,592 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,593 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 873 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,593 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,593 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,691 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:98ms, electionTimeout:98ms datanode_2 | 2019-10-15 03:16:44,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,691 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 873 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,716 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection790: begin an election at term 874 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,719 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection790 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,719 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection790 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,719 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection790: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t874, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,720 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,720 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,720 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 874 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,720 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,737 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 845 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,737 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,737 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,737 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,814 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 846 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:44,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,814 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:44,814 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:44,891 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:171ms, electionTimeout:171ms datanode_2 | 2019-10-15 03:16:44,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:44,892 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 874 for changeToCandidate datanode_2 | 2019-10-15 03:16:44,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:44,905 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection791: begin an election at term 875 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,908 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection791 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,908 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection791 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,908 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection791: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t875, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:44,908 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,908 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:44,909 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 875 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:44,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:44,909 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:45,034 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 847 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:45,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:45,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:45,034 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:45,103 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:194ms, electionTimeout:194ms datanode_2 | 2019-10-15 03:16:45,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:45,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 875 for changeToCandidate datanode_2 | 2019-10-15 03:16:45,103 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:45,116 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection792: begin an election at term 876 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:45,119 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection792 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,120 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection792 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,120 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection792: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t876, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:45,120 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,120 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,120 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 876 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:45,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:45,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:45,232 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:111ms, electionTimeout:111ms datanode_2 | 2019-10-15 03:16:45,232 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:45,232 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 876 for changeToCandidate datanode_2 | 2019-10-15 03:16:45,232 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:45,258 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection793: begin an election at term 877 for -1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:45,259 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 848 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:45,259 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:45,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:45,260 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:45,260 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection793 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection793 got exception when requesting votes: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,261 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection793: Election REJECTED; received 0 response(s) [] and 2 exception(s); 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t877, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:16:45,261 INFO impl.LeaderElection: Exception 0: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,261 INFO impl.LeaderElection: Exception 1: {} datanode_2 | java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) datanode_2 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3: group-B866F3D9F4B7 not found. datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) datanode_2 | at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) datanode_2 | at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) datanode_2 | at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99) datanode_2 | at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) datanode_2 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) datanode_2 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) datanode_2 | ... 1 more datanode_2 | 2019-10-15 03:16:45,262 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from CANDIDATE to FOLLOWER at term 877 for DISCOVERED_A_NEW_TERM datanode_2 | 2019-10-15 03:16:45,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:45,262 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:45,379 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-FollowerState: change to CANDIDATE, lastRpcTime:117ms, electionTimeout:117ms datanode_2 | 2019-10-15 03:16:45,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:45,379 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: changes role from FOLLOWER to CANDIDATE at term 877 for changeToCandidate datanode_2 | 2019-10-15 03:16:45,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:46,433 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove CANDIDATE 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7:t878, leader=null, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLog:OPENED:c-1,f-1,i0, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_2 | 2019-10-15 03:16:46,435 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: shutdown datanode_2 | 2019-10-15 03:16:46,436 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-B866F3D9F4B7,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:46,436 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:46,436 INFO impl.StateMachineUpdater: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-StateMachineUpdater: set stopIndex = -1 datanode_2 | 2019-10-15 03:16:46,437 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7: closes. applyIndex: -1 datanode_2 | 2019-10-15 03:16:46,441 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | 2019-10-15 03:16:46,441 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-SegmentedRaftLogWorker close() datanode_2 | 2019-10-15 03:16:46,513 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove LEADER 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD:t1, leader=749b19c7-0772-44d2-8efd-0664e6aa0748, voted=749b19c7-0772-44d2-8efd-0664e6aa0748, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLog:OPENED:c0,f0,i0, conf=0: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null RUNNING datanode_2 | 2019-10-15 03:16:46,513 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: shutdown datanode_2 | 2019-10-15 03:16:46,513 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-180984EA4EAD,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:46,513 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderState datanode_2 | 2019-10-15 03:16:46,514 INFO impl.PendingRequests: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-PendingRequests: sendNotLeaderResponses datanode_2 | 2019-10-15 03:16:46,514 INFO impl.StateMachineUpdater: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-StateMachineUpdater: set stopIndex = 0 datanode_2 | 2019-10-15 03:16:46,515 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD: closes. applyIndex: 0 datanode_2 | 2019-10-15 03:16:46,515 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | 2019-10-15 03:16:46,517 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-180984EA4EAD-SegmentedRaftLogWorker close() datanode_2 | 2019-10-15 03:16:49,819 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 849 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:49,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:49,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:49,819 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:49,821 WARN util.FileUtils: Failed to Files.move /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/raft-meta.tmp to /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/raft-meta: java.nio.file.NoSuchFileException: /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/raft-meta.tmp datanode_2 | 2019-10-15 03:16:49,821 WARN util.AtomicFileOutputStream: Unable to delete tmp file during abort /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/raft-meta.tmp datanode_2 | 2019-10-15 03:16:49,821 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-B866F3D9F4B7-LeaderElection794: NoSuchFileException is safely ignored since this is already CLOSING datanode_2 | java.nio.file.NoSuchFileException: /data/metadata/ratis/1f712d15-b189-4724-8627-b866f3d9f4b7/current/raft-meta.tmp datanode_2 | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) datanode_2 | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) datanode_2 | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) datanode_2 | at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:430) datanode_2 | at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:263) datanode_2 | at java.base/java.nio.file.Files.move(Files.java:1421) datanode_2 | at org.apache.ratis.util.FileUtils.lambda$move$6(FileUtils.java:65) datanode_2 | at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:50) datanode_2 | at org.apache.ratis.util.FileUtils.move(FileUtils.java:64) datanode_2 | at org.apache.ratis.util.FileUtils.move(FileUtils.java:60) datanode_2 | at org.apache.ratis.util.AtomicFileOutputStream.close(AtomicFileOutputStream.java:73) datanode_2 | at org.apache.ratis.server.storage.MetaFile.writeFile(MetaFile.java:96) datanode_2 | at org.apache.ratis.server.storage.MetaFile.set(MetaFile.java:75) datanode_2 | at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.writeMetadata(SegmentedRaftLog.java:473) datanode_2 | at org.apache.ratis.server.impl.ServerState.persistMetadata(ServerState.java:231) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:176) datanode_2 | at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) datanode_2 | at java.base/java.lang.Thread.run(Thread.java:834) datanode_2 | 2019-10-15 03:16:49,892 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 850 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:49,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:49,893 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:49,893 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,103 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 851 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,104 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,104 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,159 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 852 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,159 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,159 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,159 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,234 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 853 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,235 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,235 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,235 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 854 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,250 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,277 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 855 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,277 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,304 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 856 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,305 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,460 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 857 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,460 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,569 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 858 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,569 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,658 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 859 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,658 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,683 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 860 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,683 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,683 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,683 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,699 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 861 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,699 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,699 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,699 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 862 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,803 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:50,975 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 863 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:50,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:50,975 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:50,975 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,008 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 864 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,008 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,008 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,008 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,267 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 865 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,267 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,267 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,362 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 866 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,362 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,589 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 867 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,589 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,771 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 868 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,771 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,772 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,772 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:51,959 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 869 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:51,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:51,959 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:51,959 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,078 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 870 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,078 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,078 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,078 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,188 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 871 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,188 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,289 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 872 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,289 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 873 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,442 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,640 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 874 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,640 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,640 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,640 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 875 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:52,911 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 876 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:52,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:52,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:52,911 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,144 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 877 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:53,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:53,144 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:53,144 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,216 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 878 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:53,216 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:53,216 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:53,216 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,461 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 879 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:53,461 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:53,461 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:53,461 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,648 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 880 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:53,648 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:53,648 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:53,648 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 881 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:53,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:53,790 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:53,790 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:53,958 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: addNew group-366CBCE63B0D:[749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858] returns group-366CBCE63B0D:java.util.concurrent.CompletableFuture@35ca9f70[Not completed] datanode_2 | 2019-10-15 03:16:53,964 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748: new RaftServerImpl for group-366CBCE63B0D:[749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858] with ContainerStateMachine:uninitialized datanode_2 | 2019-10-15 03:16:53,965 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_2 | 2019-10-15 03:16:53,965 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_2 | 2019-10-15 03:16:53,965 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_2 | 2019-10-15 03:16:53,965 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_2 | 2019-10-15 03:16:53,966 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-15 03:16:53,966 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: ConfigurationManager, init=-1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null, confs= datanode_2 | 2019-10-15 03:16:53,966 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | 2019-10-15 03:16:53,967 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | 2019-10-15 03:16:53,967 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/de3123d6-eac8-4e25-8c6b-366cbce63b0d does not exist. Creating ... datanode_2 | 2019-10-15 03:16:53,980 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/de3123d6-eac8-4e25-8c6b-366cbce63b0d/in_use.lock acquired by nodename 7@b2e6e9267224 datanode_2 | 2019-10-15 03:16:54,003 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/de3123d6-eac8-4e25-8c6b-366cbce63b0d has been successfully formatted. datanode_2 | 2019-10-15 03:16:54,003 [pool-8-thread-1] INFO - group-366CBCE63B0D: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-15 03:16:54,004 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_2 | 2019-10-15 03:16:54,004 INFO segmented.SegmentedRaftLogWorker: new 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/de3123d6-eac8-4e25-8c6b-366cbce63b0d datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | 2019-10-15 03:16:54,005 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_2 | 2019-10-15 03:16:54,005 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | 2019-10-15 03:16:54,006 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_2 | 2019-10-15 03:16:54,006 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | 2019-10-15 03:16:54,006 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | 2019-10-15 03:16:54,006 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-15 03:16:54,011 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: start as a follower, conf=-1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null datanode_2 | 2019-10-15 03:16:54,011 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | 2019-10-15 03:16:54,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,012 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-366CBCE63B0D,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:16:54,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 882 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,181 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-FollowerState: change to CANDIDATE, lastRpcTime:169ms, electionTimeout:169ms datanode_2 | 2019-10-15 03:16:54,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: changes role from FOLLOWER to CANDIDATE at term 0 for changeToCandidate datanode_2 | 2019-10-15 03:16:54,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderElection datanode_2 | 2019-10-15 03:16:54,186 INFO impl.LeaderElection: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-LeaderElection795: begin an election at term 1 for -1: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null datanode_2 | 2019-10-15 03:16:54,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown LeaderElection datanode_2 | 2019-10-15 03:16:54,186 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: changes role from CANDIDATE to LEADER at term 1 for changeToLeader datanode_2 | 2019-10-15 03:16:54,187 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: change Leader from null to 749b19c7-0772-44d2-8efd-0664e6aa0748 at term 1 for becomeLeader, leader elected after 182ms datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default) datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.rpc.sleep.time = 25ms (default) datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default) datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default) datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s (default) datanode_2 | 2019-10-15 03:16:54,187 INFO server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default) datanode_2 | 2019-10-15 03:16:54,191 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start LeaderState datanode_2 | 2019-10-15 03:16:54,192 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | 2019-10-15 03:16:54,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D: set configuration 0: [749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858], old=null at 0 datanode_2 | 2019-10-15 03:16:54,226 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-366CBCE63B0D-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/de3123d6-eac8-4e25-8c6b-366cbce63b0d/current/log_inprogress_0 datanode_2 | 2019-10-15 03:16:54,272 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 883 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,272 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,329 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 884 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,329 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,329 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,329 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,438 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 885 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,438 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,438 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,453 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 886 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,453 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,453 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,482 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 887 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,482 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,482 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 888 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,611 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 889 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,704 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,753 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 890 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,753 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:54,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 891 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:54,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:54,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:54,999 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,119 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 892 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,119 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,120 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,202 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 893 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,202 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,247 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 894 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,248 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,362 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 895 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,362 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,409 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 896 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,409 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,544 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 897 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,544 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,544 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,696 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 898 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,696 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,696 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,697 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,734 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 899 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,734 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,734 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:55,926 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 900 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:55,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:55,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:55,926 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,141 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 901 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,141 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,330 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 902 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,330 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,330 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,518 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 903 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,518 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,518 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,519 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,673 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 904 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,673 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,673 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,673 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,703 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 905 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,703 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,703 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,745 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 906 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,745 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,745 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,745 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:56,934 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 907 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:56,934 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:56,934 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:56,934 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,001 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 908 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,001 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,070 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 909 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,279 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 910 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,279 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,359 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 911 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,359 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,359 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,567 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 912 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,567 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,586 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 913 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,586 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,586 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,749 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 914 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,749 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,749 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:57,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 915 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:57,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:57,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:57,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 916 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,112 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,275 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 917 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,275 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,276 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,276 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,472 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 918 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,472 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,473 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,487 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 919 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,487 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,488 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 920 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,752 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 921 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,752 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,752 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,809 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 922 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,809 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,809 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,870 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 923 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,870 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:58,968 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 924 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:58,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:58,968 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:58,968 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,096 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 925 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,096 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,306 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 926 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,306 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,498 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 927 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,498 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,498 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,569 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 928 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,569 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,569 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,795 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 929 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,795 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,796 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:16:59,962 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 930 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:16:59,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:16:59,962 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:16:59,962 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 931 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,122 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,122 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,273 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 932 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,273 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,352 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 933 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,352 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,353 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,353 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,449 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 934 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,449 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,449 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,449 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,552 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 935 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,552 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,553 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,725 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 936 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,725 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,725 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:00,870 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 937 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:00,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:00,870 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:00,870 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 938 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,108 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,257 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 939 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,257 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,257 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,399 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 940 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,399 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,399 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 941 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,477 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 942 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,550 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,576 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 943 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,576 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,763 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 944 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,764 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,764 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 945 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,792 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:01,819 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 946 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:01,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:01,819 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:01,819 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,040 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 947 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,040 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,138 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 948 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,138 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,138 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,270 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 949 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,270 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,427 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 950 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,427 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,428 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,428 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,467 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 951 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,467 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,467 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,589 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 952 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,589 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,589 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:02,794 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 953 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:02,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:02,794 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:02,794 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,001 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 954 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,001 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 955 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,178 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,409 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 956 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,409 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,410 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,410 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,623 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 957 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,623 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,623 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,851 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 958 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,851 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:03,877 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 959 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:03,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:03,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:03,878 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,016 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 960 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,017 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,017 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,117 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 961 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,117 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,117 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,117 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,287 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 962 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,288 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,288 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,383 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 963 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,383 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,383 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,418 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 964 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,418 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,419 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,419 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,497 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 965 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,497 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,497 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,649 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 966 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,649 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,649 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,740 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 967 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,741 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,741 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,859 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 968 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,859 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,859 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:04,985 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 969 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:04,985 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:04,986 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:04,986 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,062 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 970 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,062 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,101 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 971 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,101 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,101 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,226 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 972 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,226 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 973 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,393 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 974 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,393 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,611 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 975 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,611 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,799 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 976 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,799 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,799 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,812 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 977 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,812 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,812 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,813 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 978 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:05,982 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 979 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:05,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:05,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:05,982 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,041 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 980 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,041 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,188 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 981 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,189 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,189 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,290 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 982 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,531 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 983 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,531 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,531 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,532 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,628 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 984 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,628 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,628 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,747 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 985 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,747 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,747 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:06,929 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 986 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:06,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:06,929 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:06,929 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,150 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 987 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,151 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,307 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 988 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,307 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,307 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,556 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 989 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,556 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,614 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 990 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,614 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,614 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,694 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 991 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,694 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,694 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,694 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,731 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 992 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,731 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,731 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,732 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,780 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 993 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,780 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,780 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:07,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 994 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:07,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:07,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:07,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,003 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 995 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,003 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,003 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,083 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 996 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,083 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,083 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,083 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 997 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,179 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,331 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 998 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,331 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,331 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,402 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 999 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,402 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,402 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,402 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,647 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1000 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,861 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1001 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,861 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,861 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:08,979 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1002 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:08,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:08,979 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:08,979 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,070 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1003 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,070 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,208 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1004 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,208 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,208 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1005 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,298 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,397 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1006 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,397 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,397 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,489 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1007 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,489 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,697 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1008 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,697 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,728 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1009 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,728 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:09,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1010 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:09,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:09,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:09,927 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,002 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1011 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,002 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,002 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,070 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1012 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,070 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,070 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,129 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1013 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,129 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,129 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,380 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1014 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,380 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,380 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,413 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1015 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,413 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,413 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,594 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1016 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,595 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,595 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,595 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,686 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1017 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,686 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,686 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,686 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,769 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1018 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,769 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,769 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,769 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:10,990 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1019 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:10,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:10,990 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:10,990 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,075 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1020 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,075 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,076 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,271 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1021 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,272 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,272 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,290 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1022 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,290 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,290 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,464 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1023 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,464 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1024 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,601 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,655 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1025 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,655 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,655 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,865 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1026 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,865 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,931 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1027 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,931 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,931 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,931 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:11,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1028 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:11,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:11,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:11,946 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,081 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1029 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,081 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,110 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1030 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,110 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,277 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1031 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,277 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,328 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1032 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,328 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,490 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1033 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,490 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,490 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,490 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1034 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:12,853 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1035 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:12,853 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:12,853 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:12,853 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,049 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1036 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,050 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,076 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1037 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,076 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,077 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,077 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,209 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1038 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,209 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,370 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1039 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,370 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1040 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,422 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,515 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1041 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,516 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,516 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,516 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,597 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1042 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,597 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,597 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1043 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,820 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1044 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,820 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,821 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:13,925 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1045 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:13,925 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:13,926 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:13,926 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,041 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1046 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,041 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,256 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1047 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,256 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,256 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,257 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,347 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1048 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,347 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,347 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,555 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1049 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,555 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,555 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,582 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1050 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,582 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,701 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1051 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,701 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,701 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:14,888 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1052 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:14,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:14,888 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:14,888 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,095 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1053 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,095 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,196 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1054 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,196 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,299 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1055 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,299 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,300 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,300 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,488 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1056 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,488 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,711 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1057 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,711 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,712 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,712 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,879 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1058 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,879 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,879 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:15,954 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1059 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:15,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:15,954 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:15,954 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:16,048 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1060 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:16,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:16,048 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:16,048 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:16,279 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1061 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:16,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:16,279 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:16,279 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:16,469 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1062 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:16,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:16,469 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:16,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:16,653 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1063 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:16,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:16,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:16,653 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:16,848 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1064 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:16,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:16,849 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:16,849 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,060 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1065 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,060 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,060 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,060 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,157 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1066 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,157 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,157 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,350 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1067 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,403 [Command processor thread] ERROR - Container #4 does not exist in datanode. Container close failed. datanode_2 | 2019-10-15 03:17:17,403 [Command processor thread] ERROR - Container #5 does not exist in datanode. Container close failed. datanode_2 | 2019-10-15 03:17:17,566 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1068 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,567 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,567 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,591 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1069 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,591 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,591 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,687 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1070 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,687 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,687 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,738 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1071 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,738 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,738 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,805 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1072 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,805 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,805 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,845 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1073 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,845 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,845 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:17,875 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1074 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:17,875 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:17,875 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:17,875 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,025 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1075 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,026 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,026 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,179 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1076 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,179 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,179 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,323 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1077 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,323 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,323 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,415 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1078 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,415 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,553 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1079 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,553 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,642 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1080 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,642 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,659 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1081 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,659 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,659 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:18,850 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1082 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:18,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:18,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:18,850 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1083 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,060 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,235 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1084 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,235 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,235 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,235 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,452 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1085 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,452 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,452 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1086 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,563 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,633 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1087 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,633 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,633 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,774 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1088 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,774 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:19,881 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1089 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:19,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:19,881 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:19,881 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1090 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,112 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,140 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1091 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,140 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,140 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,140 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1092 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,275 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,275 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,404 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1093 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,404 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,602 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1094 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,602 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,602 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,602 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1095 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1096 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,802 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,960 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1097 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,960 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,960 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:20,987 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1098 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:20,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:20,987 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:20,987 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,079 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1099 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,079 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,080 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,080 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,229 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1100 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,229 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,229 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,258 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1101 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,258 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,258 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,478 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1102 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,478 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,550 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1103 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,550 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,550 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,740 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1104 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,740 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,740 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:21,951 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1105 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:21,951 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:21,951 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:21,951 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,038 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1106 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,038 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,038 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1107 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1108 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,315 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,480 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1109 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,480 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,480 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,480 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,662 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1110 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,662 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,855 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1111 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,855 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:22,980 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1112 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:22,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:22,980 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:22,980 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,058 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1113 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,058 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,058 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,058 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,147 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1114 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,147 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,147 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,147 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,282 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1115 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,282 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,282 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,494 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1116 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,494 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,670 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1117 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,670 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,709 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1118 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,709 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,709 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:23,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1119 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:23,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:23,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:23,949 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1120 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,059 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1121 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,240 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,240 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1122 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,463 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,691 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1123 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,691 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,692 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,692 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,864 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1124 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,864 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,864 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,904 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1125 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,904 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,904 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:24,948 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1126 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:24,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:24,949 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:24,949 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,192 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1127 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,192 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,192 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,316 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1128 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,316 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,316 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,531 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1129 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,531 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,531 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,531 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,564 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1130 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,564 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,565 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,565 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,579 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1131 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,579 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,579 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,579 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,762 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1132 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,762 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,762 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,762 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:25,821 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1133 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:25,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:25,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:25,822 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,047 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1134 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,047 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,047 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,047 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,143 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1135 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,143 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,143 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1136 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,158 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,217 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1137 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,217 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,304 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1138 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,304 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,508 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1139 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,509 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,509 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,629 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1140 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,629 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,821 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1141 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,821 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,821 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,821 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:26,973 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1142 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:26,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:26,973 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:26,973 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,072 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1143 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,072 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,072 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,142 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1144 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,142 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,348 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1145 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,348 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,348 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,416 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1146 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,416 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,416 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,460 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1147 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,460 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,477 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1148 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,477 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,477 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,524 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1149 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,524 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,524 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,578 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1150 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,578 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,578 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,667 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1151 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,667 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,667 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:27,877 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1152 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:27,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:27,877 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:27,877 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,021 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1153 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,021 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,021 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,162 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1154 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,162 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,162 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,162 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,304 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1155 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,304 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,304 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,504 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1156 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,504 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,504 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,504 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1157 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,675 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1158 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,675 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,675 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,675 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,707 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1159 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,707 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,707 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,851 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1160 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,851 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,852 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:28,852 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,903 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1161 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:28,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:28,903 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:28,903 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,125 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1162 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,125 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,125 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,220 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1163 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,220 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,484 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1164 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,484 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,484 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,499 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1165 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,499 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,499 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,566 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1166 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,566 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,566 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,678 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1167 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,678 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,678 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,678 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:29,890 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1168 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:29,890 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:29,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:29,891 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,041 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1169 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,041 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1170 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,179 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,251 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1171 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,251 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,370 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1172 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,370 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,508 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1173 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,508 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,508 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,576 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1174 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,576 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,576 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,696 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1175 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,696 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,697 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,697 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,803 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1176 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,803 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:30,992 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1177 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:30,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:30,993 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:30,993 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,052 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1178 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,052 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,053 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,120 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1179 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,120 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,120 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,263 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1180 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,263 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,263 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,277 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1181 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,277 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,277 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,306 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1182 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,306 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,306 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,424 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1183 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,424 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,424 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,491 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1184 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,491 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,491 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,684 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1185 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,684 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,684 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,684 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,840 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1186 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,840 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,841 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,841 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:31,991 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1187 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:31,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:31,991 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:31,991 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,109 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1188 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,109 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,109 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,247 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1189 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,248 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,248 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,303 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1190 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,303 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,303 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,489 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1191 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,489 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,489 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,671 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1192 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,671 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,671 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,773 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1193 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,773 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,774 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:32,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1194 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:32,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:32,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:32,954 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,124 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1195 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,124 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,124 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,124 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,212 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1196 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,212 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,212 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,443 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1197 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,444 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,444 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,629 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1198 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,629 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,629 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,850 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1199 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,850 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,850 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:33,913 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1200 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:33,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:33,913 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:33,913 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1201 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,236 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1202 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,236 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,236 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,432 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1203 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,432 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,432 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,432 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,533 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1204 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,533 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,534 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,534 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,559 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1205 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,559 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,560 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,560 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,672 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1206 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,672 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,754 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1207 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,754 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:34,803 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1208 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:34,803 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:34,804 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:34,804 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,031 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1209 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,031 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,031 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1210 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,194 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,194 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,392 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1211 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,393 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,393 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,419 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1212 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,419 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,419 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,419 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,603 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1213 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,603 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,603 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,603 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,753 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1214 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,753 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,753 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,810 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1215 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,810 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,810 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:35,847 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1216 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:35,847 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:35,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:35,848 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1217 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,006 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,158 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1218 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,158 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,158 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,186 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1219 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,186 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,186 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,373 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1220 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,373 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,373 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,553 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1221 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,553 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,553 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,676 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1222 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,676 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,677 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:36,886 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1223 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:36,886 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:36,886 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:36,886 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,040 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1224 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,040 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,041 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,041 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,190 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1225 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,190 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,321 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1226 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,321 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,322 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,478 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1227 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,478 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,478 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,685 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1228 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,685 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,685 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,685 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,788 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1229 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,788 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:37,941 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1230 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:37,941 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:37,941 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:37,941 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,136 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1231 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,136 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,270 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1232 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,270 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,270 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,287 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1233 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,287 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,463 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1234 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,463 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,463 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,632 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1235 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,632 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,632 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,800 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1236 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,800 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,800 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,800 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,848 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1237 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,848 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,848 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:38,911 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1238 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:38,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:38,911 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:38,912 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:39,081 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1239 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:39,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:39,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:39,081 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:39,297 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1240 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:39,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:39,297 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:39,297 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:39,543 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1241 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:39,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:39,543 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:39,544 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:39,610 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1242 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:39,610 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:39,610 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:39,610 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:39,836 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1243 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:39,836 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:39,836 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:39,836 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,032 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1244 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,032 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,032 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,032 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,112 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1245 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,112 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,113 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,113 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,315 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1246 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,315 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,315 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,455 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1247 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,455 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,580 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1248 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,581 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,581 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,581 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,693 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1249 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,694 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,694 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:40,785 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1250 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:40,785 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:40,785 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:40,785 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,018 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1251 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,019 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,019 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,083 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1252 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,084 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,084 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,084 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,193 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1253 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,193 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,193 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,298 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1254 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,298 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,298 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1255 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,464 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1256 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,464 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,464 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,583 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1257 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,583 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,583 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,766 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1258 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,766 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,766 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:41,891 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1259 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:41,891 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:41,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:41,892 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,071 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1260 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,071 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,071 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,141 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1261 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,141 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,141 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,231 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1262 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,231 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,231 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,260 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1263 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,260 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,403 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1264 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,404 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,563 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1265 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,563 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,563 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,723 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1266 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,723 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,723 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1267 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,802 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,855 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1268 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,855 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,855 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:42,957 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1269 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:42,957 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:42,958 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:42,958 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,116 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1270 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,116 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,116 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,362 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1271 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,362 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,362 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,502 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1272 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,503 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,503 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,653 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1273 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,653 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,680 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1274 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,680 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,680 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,680 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,769 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1275 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,770 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,770 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:43,960 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1276 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:43,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:43,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:43,961 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,170 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1277 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,170 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,170 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,170 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,375 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1278 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,375 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,375 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,404 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1279 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,404 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,505 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1280 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,506 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,506 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,547 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1281 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,547 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,547 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,547 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,754 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1282 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,754 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,754 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,760 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1283 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,760 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,760 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:44,920 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1284 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:44,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:44,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:44,920 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,062 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1285 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,062 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,062 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,166 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1286 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,166 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,166 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,205 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1287 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,205 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,250 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1288 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,250 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,250 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,459 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1289 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,459 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,459 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,459 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,658 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1290 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,658 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,658 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,658 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:45,862 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1291 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:45,862 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:45,863 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:45,863 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,009 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1292 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,009 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,010 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,010 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,087 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1293 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,087 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,087 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,273 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1294 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,273 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,273 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,338 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1295 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,338 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,338 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,492 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1296 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,492 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,492 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,492 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,523 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1297 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,523 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,523 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,630 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1298 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,630 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,630 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,630 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,747 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1299 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,748 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,748 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,748 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,892 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1300 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,892 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,892 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:46,920 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1301 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:46,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:46,920 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:46,920 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,006 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1302 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,006 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,006 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,217 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1303 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,217 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,217 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,396 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1304 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,396 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,396 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,514 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1305 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,514 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,514 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,600 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1306 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,600 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,601 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,601 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:47,830 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1307 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:47,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:47,830 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:47,830 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,000 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1308 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,001 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,001 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,209 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1309 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,209 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,210 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,210 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,325 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1310 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,325 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,325 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,414 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1311 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,415 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,415 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,454 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1312 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,455 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,455 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,672 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1313 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,672 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,672 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,789 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1314 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,789 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,789 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:48,857 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1315 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:48,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:48,858 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:48,858 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,034 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1316 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,034 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,034 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,196 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1317 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,196 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,196 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,247 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1318 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,247 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,247 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,339 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1319 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,339 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,339 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,354 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1320 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,354 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,354 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,462 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1321 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,462 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,462 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,615 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1322 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,615 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,615 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,615 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,694 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1323 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,694 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,694 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,694 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,726 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1324 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,726 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,726 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,726 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:49,949 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1325 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:49,950 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:49,950 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:49,950 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,175 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1326 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,175 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,175 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,403 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1327 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,403 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,403 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,476 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1328 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,476 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,626 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1329 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,626 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,626 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,651 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1330 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,651 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,651 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,779 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1331 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,779 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,779 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:50,956 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1332 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:50,956 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:50,956 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:50,957 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,011 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1333 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,011 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,012 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,012 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,065 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1334 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,065 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,065 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,260 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1335 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,260 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,260 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,295 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1336 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,295 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,295 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,404 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1337 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,404 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,405 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,405 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,476 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1338 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,476 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,476 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,587 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1339 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,587 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,587 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,785 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1340 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,785 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,786 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,786 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,955 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1341 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,955 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,955 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:51,999 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1342 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:51,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:51,999 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:51,999 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,081 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1343 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,081 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,081 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,096 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1344 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,096 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,096 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,226 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1345 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,226 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,226 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,379 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1346 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,379 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,380 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,501 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1347 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,501 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,501 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,714 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1348 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,714 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,714 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:52,889 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1349 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:52,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:52,889 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:52,889 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,127 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1350 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,127 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,127 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,127 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,251 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1351 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,251 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,251 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,328 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1352 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,328 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,422 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1353 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,422 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,422 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,593 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1354 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,593 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,593 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,593 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,608 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1355 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,608 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,608 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,609 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,756 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1356 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,756 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,756 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,756 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:53,927 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1357 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:53,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:53,927 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:53,927 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,057 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1358 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,057 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,057 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,187 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: addNew group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] returns group-BD8D864B3B3A:java.util.concurrent.CompletableFuture@165d18a[Not completed] datanode_2 | 2019-10-15 03:17:54,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748: new RaftServerImpl for group-BD8D864B3B3A:[5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858] with ContainerStateMachine:uninitialized datanode_2 | 2019-10-15 03:17:54,191 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.min = 5s (custom) datanode_2 | 2019-10-15 03:17:54,191 INFO server.RaftServerConfigKeys: raft.server.rpc.timeout.max = 5200ms (custom) datanode_2 | 2019-10-15 03:17:54,191 INFO server.RaftServerConfigKeys: raft.server.rpcslowness.timeout = 120s (custom) datanode_2 | 2019-10-15 03:17:54,191 INFO server.RaftServerConfigKeys: raft.server.sleep.deviation.threshold = 300 (default) datanode_2 | 2019-10-15 03:17:54,191 INFO server.RaftServerConfigKeys: raft.server.log.appender.install.snapshot.enabled = false (custom) datanode_2 | 2019-10-15 03:17:54,191 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: ConfigurationManager, init=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null, confs= datanode_2 | 2019-10-15 03:17:54,192 INFO server.RaftServerConfigKeys: raft.server.storage.dir = [/data/metadata/ratis] (custom) datanode_2 | 2019-10-15 03:17:54,192 INFO server.RaftServerConfigKeys: raft.server.log.corruption.policy = EXCEPTION (default) datanode_2 | 2019-10-15 03:17:54,192 INFO storage.RaftStorageDirectory: The storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a does not exist. Creating ... datanode_2 | 2019-10-15 03:17:54,201 INFO storage.RaftStorageDirectory: Lock on /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/in_use.lock acquired by nodename 7@b2e6e9267224 datanode_2 | 2019-10-15 03:17:54,214 INFO storage.RaftStorage: Storage directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a has been successfully formatted. datanode_2 | 2019-10-15 03:17:54,214 [pool-8-thread-1] INFO - group-BD8D864B3B3A: The snapshot info is null. Setting the last applied indexto:(t:0, i:~) datanode_2 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.notification.no-leader.timeout = 120s (custom) datanode_2 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.use.memory = false (default) datanode_2 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.purge.gap = 1000000 (custom) datanode_2 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.appender.buffer.byte-limit = 33554432 (custom) datanode_2 | 2019-10-15 03:17:54,215 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.segment.cache.num.max = 2 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO segmented.SegmentedRaftLogWorker: new 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A-SegmentedRaftLogWorker for RaftStorage:Storage Directory /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.queue.byte-limit = 2147483647 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.queue.element-limit = 1024 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.segment.size.max = 1048576 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.preallocated.size = 16384 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.write.buffer.size = 33554432 (custom) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.force.sync.num = 128 (default) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync = true (default) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout = 10s (default) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.sync.timeout.retry = -1 (default) datanode_2 | 2019-10-15 03:17:54,216 INFO server.RaftServerConfigKeys: raft.server.log.statemachine.data.caching.enabled = true (custom) datanode_2 | 2019-10-15 03:17:54,217 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0 -> -1 datanode_2 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.enabled = true (custom) datanode_2 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.auto.trigger.threshold = 10000 (custom) datanode_2 | 2019-10-15 03:17:54,217 INFO server.RaftServerConfigKeys: raft.server.snapshot.retention.file.num = 5 (custom) datanode_2 | 2019-10-15 03:17:54,218 INFO server.RaftServerConfigKeys: raft.server.retrycache.expirytime = 600000ms (custom) datanode_2 | 2019-10-15 03:17:54,221 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: start as a follower, conf=-1: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null datanode_2 | 2019-10-15 03:17:54,221 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: changes role from null to FOLLOWER at term 0 for startAsFollower datanode_2 | 2019-10-15 03:17:54,221 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,222 INFO util.JmxRegister: Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-BD8D864B3B3A,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:17:54,278 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: changes role from FOLLOWER to FOLLOWER at term 1 for recognizeCandidate:cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 datanode_2 | 2019-10-15 03:17:54,278 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,278 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,278 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,296 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1359 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,296 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,296 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,296 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,328 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1360 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,328 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,328 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,336 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: change Leader from null to cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3 at term 1 for appendEntries, leader elected after 120ms datanode_2 | 2019-10-15 03:17:54,355 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A: set configuration 0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null at 0 datanode_2 | 2019-10-15 03:17:54,355 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A-SegmentedRaftLogWorker: Starting segment from index:0 datanode_2 | 2019-10-15 03:17:54,385 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-BD8D864B3B3A-SegmentedRaftLogWorker: created new log segment /data/metadata/ratis/9b465122-6196-4503-b052-bd8d864b3b3a/current/log_inprogress_0 datanode_2 | 2019-10-15 03:17:54,460 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1361 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,460 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,460 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,494 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1362 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,494 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,494 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,713 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1363 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,713 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,714 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:54,910 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1364 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:54,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:54,910 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:54,910 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,121 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1365 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,121 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,121 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,335 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1366 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,335 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,335 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,381 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1367 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,381 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,381 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,533 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1368 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,533 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,533 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,533 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,653 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1369 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,653 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,653 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,728 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1370 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,728 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,728 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:55,844 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1371 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:55,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:55,844 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:55,844 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,059 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1372 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,059 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,059 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,108 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1373 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,108 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,108 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,280 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1374 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,280 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,280 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,425 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1375 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,425 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,425 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,622 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1376 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,622 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,622 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,622 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:56,945 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1377 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:56,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:56,945 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:56,945 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,066 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1378 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,066 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,066 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,091 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1379 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,091 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,091 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,091 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,148 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1380 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,148 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,148 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,340 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1381 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,340 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,340 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,385 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1382 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,385 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,385 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,385 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,622 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1383 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,622 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,622 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,622 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,802 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1384 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,802 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,802 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:57,946 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1385 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:57,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:57,946 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:57,946 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,050 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1386 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,050 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,050 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,239 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1387 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,239 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,239 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,343 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1388 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,343 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,343 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,488 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1389 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,488 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,488 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,635 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1390 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,635 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,635 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,693 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1391 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,693 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,693 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,806 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1392 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,806 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,806 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,806 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:58,865 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1393 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:58,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:58,865 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:58,865 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,032 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1394 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,032 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,033 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,033 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,202 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1395 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,202 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,202 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,369 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1396 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,369 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,369 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,386 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1397 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,387 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,387 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,545 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1398 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,545 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,545 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,545 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,610 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1399 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,610 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,611 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,611 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,640 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1400 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,640 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,640 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,640 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,838 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1401 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,838 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,838 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,838 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,876 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1402 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,876 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,876 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:17:59,935 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1403 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:17:59,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:17:59,935 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:17:59,935 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,136 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1404 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,136 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,136 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,286 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1405 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,286 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,287 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,287 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,435 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1406 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,435 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,435 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,435 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,473 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1407 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,473 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,473 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,512 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1408 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,512 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,512 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,512 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,551 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1409 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,551 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,551 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,763 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1410 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,763 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,763 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:00,940 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1411 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:00,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:00,940 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:00,940 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,100 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1412 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,100 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,100 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,100 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,181 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1413 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,181 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,181 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,274 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1414 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,274 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,274 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,313 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1415 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,314 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,314 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,314 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,470 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1416 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,470 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,470 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,588 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1417 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,588 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,588 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,588 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,603 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1418 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,603 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,604 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,604 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,792 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1419 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,792 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,792 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:01,897 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1420 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:01,897 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:01,897 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:01,897 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,053 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1421 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,053 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,095 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1422 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,095 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,095 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,219 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1423 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,219 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,220 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,220 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,349 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1424 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,349 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,349 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,442 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1425 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,442 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,443 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,443 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,642 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1426 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,642 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,642 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:02,821 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1427 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:02,821 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:02,822 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:02,822 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,053 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1428 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,053 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,053 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,142 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1429 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,142 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,142 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,225 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1430 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,225 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,225 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,240 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1431 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,241 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,241 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,392 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1432 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,392 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,393 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,619 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1433 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,619 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,619 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,660 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1434 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,660 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,660 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,788 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1435 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,788 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,788 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:03,832 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1436 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:03,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:03,832 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:03,832 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,035 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1437 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,035 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,035 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,149 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1438 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,150 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,150 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,169 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1439 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,169 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,169 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,169 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,333 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1440 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,333 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,333 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,479 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1441 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,479 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,479 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,637 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1442 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,637 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,637 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,767 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1443 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,767 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,767 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,767 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:04,982 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1444 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:04,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:04,982 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:04,983 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,110 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1445 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,110 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,110 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,190 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1446 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,190 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,190 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,205 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1447 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,205 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,205 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,370 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1448 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,370 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,370 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,427 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1449 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,427 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,427 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,427 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,556 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1450 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,556 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,556 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,647 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1451 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,647 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,647 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,662 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1452 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,662 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,662 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,677 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1453 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,677 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,677 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,751 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1454 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,751 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,751 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:05,961 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1455 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:05,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:05,961 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:05,961 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,042 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1456 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,042 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,043 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,043 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,105 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1457 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,105 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,106 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,106 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,188 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1458 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,188 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,188 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,350 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1459 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,350 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,350 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,568 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1460 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,568 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,568 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,568 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,705 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1461 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,705 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,705 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,705 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,880 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1462 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,880 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,880 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,880 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:06,953 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1463 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:06,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:06,953 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:06,953 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:07,090 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: changes role from FOLLOWER to FOLLOWER at term 1464 for recognizeCandidate:5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c datanode_2 | 2019-10-15 03:18:07,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:07,090 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: start FollowerState datanode_2 | 2019-10-15 03:18:07,090 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:07,178 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove FOLLOWER 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13:t1464, leader=null, voted=null, raftlog=749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLog:OPENED:c0,f0,i0, conf=0: [5ce55bf6-dbcc-40fb-8fb4-6e78032f4b8c:172.18.0.4:9858, 749b19c7-0772-44d2-8efd-0664e6aa0748:172.18.0.5:9858, cabaa710-fa39-4ccb-a1c7-dafe5a6da4e3:172.18.0.6:9858], old=null RUNNING datanode_2 | 2019-10-15 03:18:07,178 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: shutdown datanode_2 | 2019-10-15 03:18:07,178 INFO util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-D5F35BC43A13,id=749b19c7-0772-44d2-8efd-0664e6aa0748 datanode_2 | 2019-10-15 03:18:07,178 INFO impl.RoleInfo: 749b19c7-0772-44d2-8efd-0664e6aa0748: shutdown FollowerState datanode_2 | 2019-10-15 03:18:07,178 INFO impl.StateMachineUpdater: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-StateMachineUpdater: set stopIndex = 0 datanode_2 | 2019-10-15 03:18:07,178 INFO impl.FollowerState: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted datanode_2 | 2019-10-15 03:18:07,179 INFO impl.RaftServerImpl: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13: closes. applyIndex: 0 datanode_2 | 2019-10-15 03:18:07,179 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. datanode_2 | 2019-10-15 03:18:07,181 INFO segmented.SegmentedRaftLogWorker: 749b19c7-0772-44d2-8efd-0664e6aa0748@group-D5F35BC43A13-SegmentedRaftLogWorker close() datanode_2 | 2019-10-15 03:18:37,159 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove group-D5F35BC43A13:null datanode_2 | 2019-10-15 03:19:01,661 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove group-D5F35BC43A13:null datanode_2 | 2019-10-15 03:19:03,040 INFO impl.RaftServerProxy: 749b19c7-0772-44d2-8efd-0664e6aa0748: remove group-D5F35BC43A13:null